# Table of Contents - [Posts](#posts) - [Q1 2025 tidymodels digest](#q1-2025-tidymodels-digest) - [Categories](#categories) - [Tags](#tags) - [tidymodels](#tidymodels) - [roundup](#roundup) - [Tidyverse](#tidyverse) - [Air, an extremely fast R formatter](#air-an-extremely-fast-r-formatter) - [Three experiments in LLM code assist with RStudio and Positron](#three-experiments-in-llm-code-assist-with-rstudio-and-positron) - [programming](#programming) - [ellmer](#ellmer) - [nanoparquet 0.4.0](#nanoparquet-0-4-0) - [ai](#ai) - [httr2](#httr2) - [package](#package) - [parquet](#parquet) - [graphics](#graphics) - [marquee](#marquee) - [httr2 1.1.0](#httr2-1-1-0) - [systemfonts](#systemfonts) - [textshaping](#textshaping) - [orbital](#orbital) - [Updates to Text Rendering in R Graphics](#updates-to-text-rendering-in-r-graphics) - [orbital 0.3.0](#orbital-0-3-0) - [internship](#internship) - [ggplot2](#ggplot2) - [Joining the ggplot2 team](#joining-the-ggplot2-team) - [tidymodels Internship for 2025](#tidymodels-internship-for-2025) - [wasm](#wasm) - [WebAssembly roundup part 2: Shinylive 0.8.0](#webassembly-roundup-part-2-shinylive-0-8-0) - [webr](#webr) - [shiny](#shiny) - [s7](#s7) - [S7 0.2.0](#s7-0-2-0) - [other](#other) - [postprocessing](#postprocessing) - [quarto](#quarto) - [workflows](#workflows) - [shinylive](#shinylive) - [WebAssembly roundup part 3: Quarto Live 0.1.1](#webassembly-roundup-part-3-quarto-live-0-1-1) - [webassembly](#webassembly) - [patchwork](#patchwork) - [gt](#gt) - [patchwork 1.3.0](#patchwork-1-3-0) - [WebAssembly roundup part 1: webR 0.4.2](#webassembly-roundup-part-1-webr-0-4-2) - [Postprocessing is coming to tidymodels](#postprocessing-is-coming-to-tidymodels) - [devtools](#devtools) - [recipes](#recipes) - [pkgdown 2.1.0](#pkgdown-2-1-0) - [pkgdown](#pkgdown) - [recipes 1.1.0](#recipes-1-1-0) - [bonsai](#bonsai) - [parsnip](#parsnip) - [parallelism](#parallelism) - [bonsai 0.3.0](#bonsai-0-3-0) - [censored](#censored) - [workflowsets](#workflowsets) - [Tidyverse developer day 2024](#tidyverse-developer-day-2024) - [dbplyr](#dbplyr) - [tune](#tune) - [Q1 2024 tidymodels digest](#q1-2024-tidymodels-digest) - [marquee 0.1.0](#marquee-0-1-0) - [tune 1.2.0](#tune-1-2-0) - [dbplyr 2.5.0](#dbplyr-2-5-0) - [tidyverse-dev-day](#tidyverse-dev-day) - [nanoparquet 0.3.0](#nanoparquet-0-3-0) - [Survival analysis for time-to-event data with tidymodels](#survival-analysis-for-time-to-event-data-with-tidymodels) - [yardstick](#yardstick) - [learn](#learn) - [Fair machine learning with tidymodels](#fair-machine-learning-with-tidymodels) - [Take the tidymodels survey for 2024 priorities](#take-the-tidymodels-survey-for-2024-priorities) - [ggplot2 3.5.0: Introducing: coord_radial()](#ggplot2-3-5-0-introducing-coord-radial-) - [webR 0.3.1](#webr-0-3-1) - [ggplot2-3-5-0](#ggplot2-3-5-0) - [ggplot2 3.5.0: Legends](#ggplot2-3-5-0-legends) - [survey](#survey) - [bigrquery](#bigrquery) - [ggplot2 3.5.0: Axes](#ggplot2-3-5-0-axes) - [withr](#withr) - [bigrquery 1.5.0](#bigrquery-1-5-0) - [roxygen2](#roxygen2) - [r-lib](#r-lib) - [httr](#httr) - [package maintenance](#package-maintenance) - [scales](#scales) - [ggplot2 3.5.0](#ggplot2-3-5-0) - [roxygen2 7.3.0](#roxygen2-7-3-0) - [Three ways errors are about to get better in tidymodels](#three-ways-errors-are-about-to-get-better-in-tidymodels) - [dbplyr 2.4.0](#dbplyr-2-4-0) - [testthat](#testthat) - [dplyr](#dplyr) - [Q4 2023 tidymodels digest](#q4-2023-tidymodels-digest) - [scales 1.3.0](#scales-1-3-0) - [withr 3.0.0](#withr-3-0-0) - [rsample](#rsample) - [tidyclust](#tidyclust) - [Q3 2023 tidymodels digest](#q3-2023-tidymodels-digest) - [httr2 1.0.0](#httr2-1-0-0) - [pak 0.6.0](#pak-0-6-0) - [teaching](#teaching) - [testthat 3.2.0](#testthat-3-2-0) - [tidyverse](#tidyverse) - [New interface to validation splits](#new-interface-to-validation-splits) - [deep-dive](#deep-dive) - [book](#book) - [webR 0.2.0 has been released](#webr-0-2-0-has-been-released) - [`purrr::walk()` this way](#-purrr-walk-this-way) - [R for Data Science, 2nd edition](#r-for-data-science-2nd-edition) - [gmailr](#gmailr) - [gargle](#gargle) - [spring cleaning](#spring-cleaning) - [Solutions for R4DS, 2e with Data Trail](#solutions-for-r4ds-2e-with-data-trail) - [Tidyteam code review principles](#tidyteam-code-review-principles) - [Q2 2023 tidymodels digest](#q2-2023-tidymodels-digest) - [gmailr 2.0.0](#gmailr-2-0-0) - [Package spring cleaning](#package-spring-cleaning) - [purrr](#purrr) - [desirability](#desirability) - [desirability2](#desirability2) - [optimization](#optimization) - [magrittr](#magrittr) - [Teaching the tidyverse in 2023](#teaching-the-tidyverse-in-2023) - [fs](#fs) - [Q1 2023 tidymodels digest](#q1-2023-tidymodels-digest) - [package](#package) - [Differences between the base R and magrittr pipes](#differences-between-the-base-r-and-magrittr-pipes) - [censored 0.2.0](#censored-0-2-0) - [Tuning hyperparameters with tidymodels is a delight](#tuning-hyperparameters-with-tidymodels-is-a-delight) - [vctrs](#vctrs) - [dials](#dials) - [dplyr 1.1.1](#dplyr-1-1-1) - [dtplyr](#dtplyr) - [New CRAN requirements for packages with C and C++](#new-cran-requirements-for-packages-with-c-and-c-) - [tidyverse 2.0.0](#tidyverse-2-0-0) - [dplyr 1.1.0: The power of vctrs](#dplyr-1-1-0-the-power-of-vctrs) - [dplyr-1-1-0](#dplyr-1-1-0) - [Writing performant code with tidy tools](#writing-performant-code-with-tidy-tools) - [dplyr 1.1.0: Joins](#dplyr-1-1-0-joins) - [dtplyr 1.3.0](#dtplyr-1-3-0) - [forcats](#forcats) - [webR 0.1.0 has been released](#webr-0-1-0-has-been-released) - [dplyr 1.1.0: `pick()`, `reframe()`, and `arrange()`](#dplyr-1-1-0-pick-reframe-and-arrange-) - [tidyr 1.3.0](#tidyr-1-3-0) - [dplyr 1.1.0: Per-operation grouping](#dplyr-1-1-0-per-operation-grouping) - [tidyr](#tidyr) - [dbplyr 2.3.0](#dbplyr-2-3-0) - [forcats 1.0.0](#forcats-1-0-0) - [Q4 2022 tidymodels digest](#q4-2022-tidymodels-digest) - [Model Calibration](#model-calibration) - [stringr](#stringr) - [plots](#plots) - [tidyclust is on CRAN](#tidyclust-is-on-cran) - [stringr 1.5.0](#stringr-1-5-0) - [agua](#agua) - [purrr 1.0.0](#purrr-1-0-0) - [ggplot2 3.4.0](#ggplot2-3-4-0) - [tidyselect](#tidyselect) - [Q3 2022 tidymodels digest](#q3-2022-tidymodels-digest) - [model](#model) - [h2o](#h2o) - [lifecycle](#lifecycle) - [clock](#clock) --- # Posts [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Posts [![](/blog/2025/02/tidymodels-2025-q1/thumbnail-sq.jpg)](/blog/2025/02/tidymodels-2025-q1/) [Q1 2025 tidymodels digest](/blog/2025/02/tidymodels-2025-q1/) [roundup](/categories/roundup) Max Kuhn A summary of the goings on for the tidymodels group in later 2024 and early 2025. [Read more ...](/blog/2025/02/tidymodels-2025-q1/) 2025/02/27 [![](/blog/2025/02/air/thumbnail-sq.jpg)](/blog/2025/02/air/) [Air, an extremely fast R formatter](/blog/2025/02/air/) [programming](/categories/programming) Davis Vaughan and Lionel Henry We are thrilled to announce Air, a new R formatter. [Read more ...](/blog/2025/02/air/) 2025/02/21 [![](/blog/2025/01/experiments-llm/thumbnail-sq.jpg)](/blog/2025/01/experiments-llm/) [Three experiments in LLM code assist with RStudio and Positron](/blog/2025/01/experiments-llm/) [roundup](/categories/roundup) Simon Couch We’ve been experimenting with LLM-powered tools to streamline R data science and package development. [Read more ...](/blog/2025/01/experiments-llm/) 2025/01/29 [![](/blog/2025/01/nanoparquet-0-4-0/thumbnail-sq.jpg)](/blog/2025/01/nanoparquet-0-4-0/) [nanoparquet 0.4.0](/blog/2025/01/nanoparquet-0-4-0/) [package](/categories/package) Gábor Csárdi nanoparquet 0.4.0 comes with a new and much faster `read_parquet()`, configurable type mappings in `write_parquet()`, and a new `append_parquet()`. [Read more ...](/blog/2025/01/nanoparquet-0-4-0/) 2025/01/28 [![](/blog/2025/01/httr2-1-1-0/thumbnail-sq.jpg)](/blog/2025/01/httr2-1-1-0/) [httr2 1.1.0](/blog/2025/01/httr2-1-1-0/) [package](/categories/package) Hadley Wickham httr2 1.1.0 introduces powerful new streaming capabilities with `req_perform_connection()`, as well as comprehensive URL manipulation tools, improved AWS support, and a bunch of bug fixes. [Read more ...](/blog/2025/01/httr2-1-1-0/) 2025/01/20 [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 [![](/blog/2025/01/orbital-0-3-0/thumbnail-sq.jpg)](/blog/2025/01/orbital-0-3-0/) [orbital 0.3.0](/blog/2025/01/orbital-0-3-0/) [package](/categories/package) Emil Hvitfeldt orbital 0.3.0 is on CRAN! orbital now has classification support. [Read more ...](/blog/2025/01/orbital-0-3-0/) 2025/01/13 [![](/blog/2025/01/joining-ggplot2/thumbnail-sq.jpg)](/blog/2025/01/joining-ggplot2/) [Joining the ggplot2 team](/blog/2025/01/joining-ggplot2/) [other](/categories/other) Teun van den Brand I joined the ggplot2 team and would like to share the experience. [Read more ...](/blog/2025/01/joining-ggplot2/) 2025/01/09 [![](/blog/2025/01/tidymodels-2025-internship/thumbnail-sq.jpg)](/blog/2025/01/tidymodels-2025-internship/) [tidymodels Internship for 2025](/blog/2025/01/tidymodels-2025-internship/) [other](/categories/other) Max Kuhn The tidymodels team is sponsoring a summer internship in 2025. [Read more ...](/blog/2025/01/tidymodels-2025-internship/) 2025/01/08 [![](/blog/2024/11/s7-0-2-0/thumbnail-sq.jpg)](/blog/2024/11/s7-0-2-0/) [S7 0.2.0](/blog/2024/11/s7-0-2-0/) [package](/categories/package) Tomasz Kalinowski and Hadley Wickham S7 is a new package that simplifies object-oriented programming (OOP) in R. It combines the simplicity of S3 with the structure of S4 to create a clearer system that’s accessible to everyone. [Read more ...](/blog/2024/11/s7-0-2-0/) 2024/11/07 * [««](/blog/) * « * [1](/blog/) * [2](/blog/page/2/) * [3](/blog/page/3/) *  …  * [32](/blog/page/32/) * [»](/blog/page/2/) * [»»](/blog/page/32/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q1 2025 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q1 2025 tidymodels digest ========================= ![](/blog/2025/02/tidymodels-2025-q1/thumbnail-wd.jpg) Photo by George Stagg and Max Kuhn   2025/02/27   [tidymodels](/tags/tidymodels/)   Max Kuhn The tidymodels framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing quarterly updates here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the tidymodels tag to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused. We’ve sent a steady stream of tidymodels packages to CRAN recently. We usually release them in batches since many of our packages are tightly coupled with one another. Internally, this process is referred to as the “cascade” of CRAN submissions. The post will update you on which packages have changed and the major improvements you should know about. Here’s a list of the packages and their News sections: * [baguette](https://baguette.tidymodels.org/news/index.html) * [brulee](https://brulee.tidymodels.org/news/index.html) * [censored](https://censored.tidymodels.org/news/index.html) * [dials](https://dials.tidymodels.org/news/index.html) * [hardhat](https://hardhat.tidymodels.org/news/index.html) * [parsnip](https://parsnip.tidymodels.org/news/index.html) * [recipes](https://recipes.tidymodels.org/news/index.html) * [tidymodels](https://tidymodels.tidymodels.org/news/index.html) * [tune](https://tune.tidymodels.org/news/index.html) * [workflows](https://workflows.tidymodels.org/news/index.html) Let’s look at a few specific updates. Improvements in errors and warnings[](#improvements-in-errors-and-warnings) ---------------------------------------------------------------------------- A group effort was made to improve our error and warning messages across many packages. This started with an internal “upkeep week” (which ended up being 3-4 weeks) and concluded at the [Tidy Dev Day in Seattle](https://www.tidyverse.org/blog/2024/04/tdd-2024/) after posit::conf(2024). The goal was to use new tools in the cli and rlang packages to make messages more informative than they used to be. For example, using: tidy(pca_extract_trained, number = 3, type = "variances") used to result in the error message: Error in `match.arg()`: ! 'arg' should be one of "coef", "variance" The new system references the function that you called and not the underlying base R function that actually errored. It also suggests a solution: Error in `tidy()`: ! `type` must be one of "coef" or "variance", not "variances". i Did you mean "variance"? The rlang package created a set of [standalone files](https://usethis.r-lib.org/reference/use_standalone.html) that contain high-quality type checkers and related functions. This also improves the information that users get from an error. For example, using an inappropriate formula value in `fit(linear_reg(), "boop", mtcars)`, the old message was: Error in `fit()`: ! The `formula` argument must be a formula, but it is a . and now you see: Error in `fit()`: ! `formula` must be a formula, not the string "boop". This was _a lot_ of work and we’re still aren’t finished. Two events helped us get as far as we did. First, Simon Couch made the [chores](https://simonpcouch.github.io/chores/) package (its previous name was “pal”), which enabled us to use AI tools to solve small-scope problems, such as converting old rlang error code to use the new [cli syntax](https://rlang.r-lib.org/reference/topic-condition-formatting.html) . I can’t overstate how much of a speed-up this was for us. Second, at developer day, many external folks pitched in to make pull requests from a list of issues: ![Organizing Tidy Dev Day issues.](IMG_4743.jpeg) Organizing Tidy Dev Day issues. I love these sessions for many reasons, but mostly because we meet users and contributors to our packages in person and work with them on specific tasks. There is a lot more to do here; we have a lot of secondary packages that would benefit from these improvements too. Quantile regression in parsnip[](#quantile-regression-in-parsnip) ------------------------------------------------------------------ One big update in parsnip was a new modeling mode of `"quantile regression"`. Daniel McDonald and Ryan Tibshirani largely provided some inertia for this work based on their [disease modeling framework](https://delphi.cmu.edu/) . You can generate quantile predictions by first creating a model specification, which includes the quantiles that you want to predict: library(tidymodels) tidymodels_prefer() ames <- modeldata::ames |> mutate(Sale_Price = log10(Sale_Price)) |> select(Sale_Price, Latitude) quant_spec <- linear_reg() |> set_engine("quantreg") |> set_mode("quantile regression", quantile_levels = c(0.1, 0.5, 0.9)) quant_spec ## Linear Regression Model Specification (quantile regression) ## ## Computational engine: quantreg ## Quantile levels: 0.1, 0.5, and 0.9. We’ll add some spline terms via a recipe and fit the model: spline_rec <- recipe(Sale_Price ~ ., data = ames) |> step_spline_natural(Latitude, deg_free = 10) quant_fit <- workflow(spline_rec, quant_spec) |> fit(data = ames) quant_fit ## ══ Workflow [trained] ═════════════════════════════════════════════════ ## Preprocessor: Recipe ## Model: linear_reg() ## ## ── Preprocessor ─────────────────────────────────────────────────────── ## 1 Recipe Step ## ## • step_spline_natural() ## ## ── Model ────────────────────────────────────────────────────────────── ## Call: ## quantreg::rq(formula = ..y ~ ., tau = quantile_levels, data = data) ## ## Coefficients: ## tau= 0.1 tau= 0.5 tau= 0.9 ## (Intercept) 4.71981123 5.07728741 5.25221335 ## Latitude_01 1.22409173 0.70928577 0.79000849 ## Latitude_02 0.19561816 0.04937750 0.02832633 ## Latitude_03 0.16616065 0.02045910 0.14730573 ## Latitude_04 0.30583648 0.08489487 0.15595080 ## Latitude_05 0.21663212 0.02016258 -0.01110625 ## Latitude_06 0.33541228 0.12005254 0.03006777 ## Latitude_07 0.47732205 0.09146728 0.17394021 ## Latitude_08 0.24028784 0.30450058 0.26144584 ## Latitude_09 0.05840312 -0.14733781 -0.11911843 ## Latitude_10 1.52800673 0.95994216 1.21750501 ## ## Degrees of freedom: 2930 total; 2919 residual For prediction, tidymodels always returns a data frame with as many rows as the input data set (here: `ames`). The result for quantile predictions is a special vctrs class: quant_pred <- predict(quant_fit, ames) quant_pred |> slice(1:4) ## # A tibble: 4 × 1 ## .pred_quantile ## ## 1 [5.33] ## 2 [5.33] ## 3 [5.33] ## 4 [5.31] class(quant_pred$.pred_quantile) ## [1] "quantile_pred" "vctrs_vctr" "list" where the output `[5.31]` shows the middle quantile. We can expand the set of quantile predictions so that there are three rows for each source row in `ames`. There’s also an integer column called `.row` so that we can merge the data with the source data: quant_pred$.pred_quantile[1] ## ## [1] [5.33] ## # Quantile levels: 0.1 0.5 0.9 as_tibble(quant_pred$.pred_quantile[1]) ## # A tibble: 3 × 3 ## .pred_quantile .quantile_levels .row ## ## 1 5.08 0.1 1 ## 2 5.33 0.5 1 ## 3 5.52 0.9 1 Here are the predicted quantile values: quant_pred$.pred_quantile |> as_tibble() |> full_join(ames |> add_rowindex(), by = ".row") |> arrange(Latitude) |> ggplot(aes(x = Latitude)) + geom_point(data = ames, aes(y = Sale_Price), alpha = 1 / 5) + geom_line(aes(y = .pred_quantile, col = format(.quantile_levels)), show.legend = FALSE, linewidth = 1.5) ![10%, 50%, and 90% quantile predictions.](figure/quant-plot-1.svg) 10%, 50%, and 90% quantile predictions. For now, the new mode does not have many engines. We need to implement some performance statistics in the yardstick package before integrating these models into the whole tidymodels ecosystem. In other news, we’ve added some additional neural network models based on some improvements in the brulee package. Namely, two-layer networks can be tuned for feed-forward networks on tabular data (using torch). One other improvement has been simmering for a long time: the ability to exploit sparse data structures better. We’ve improved our `fit()` interfaces for the few model engines that can use sparsely encoded data. There is much more to come on this in a few months, especially around recipes, so stay tuned. Finally, we’ve created a set of [checklists](https://parsnip.tidymodels.org/articles/checklists.html) that can be used when creating new models or engines. These are very helpful, even for us, since there is a lot of minutiae to remember. Parallelism in tune[](#parallelism-in-tune) -------------------------------------------- This was a small maintenance release mostly related to parallel processing. Up to now, tune facilitated parallelism using the [foreach](https://cran.r-project.org/package=foreach) package. That package is mature but not actively developed, so we have been slowly moving toward using the [future](https://www.futureverse.org/packages-overview.html) package(s). The [first step in this journey](https://www.tidyverse.org/blog/2024/04/tune-1-2-0/#modernized-support-for-parallel-processing) was to keep using foreach internally (but lean toward future) but to encourage users to move from directly invoking the foreach package and, instead, load and use the future package. We’re now moving folks into the second stage. tune will now raise a warning when: * A parallel backend has been registered with foreach, and * No [`plan()`](https://future.futureverse.org/reference/plan.html) has been specified with future. This will allow users to transition their existing code to only future and allow us to update existing documentation and training materials. We anticipate that the third stage, **removing foreach entirely**, will occur sometime before posit::conf(2025) in September. Things to look forward to[](#things-to-look-forward-to) -------------------------------------------------------- We are working hard on a few major initiatives that we plan on showing off at [posit::conf(2025)](https://posit.co/conference/) . First is integrated support for sparse **data**. The emphasis is on “data” because users can use a data frame of sparse vectors _or_ the usual sparse matrix format. This is a big deal because it does not force you to convert non-numeric data into a numeric matrix format. Again, we’ll discuss this more in the future, but you should be able to use sparse data frames in parsnip, recipes, tune, etc. The second initiative is the longstanding goal of adding **postprocessing** to tidymodels. Just as you can add a preprocessor to a model workflow, you will be able to add a set of postprocessing adjustments to the predictions your model generates. See our [previous post](https://www.tidyverse.org/blog/2024/10/postprocessing-preview/) for a sneak peek. Finally, this year’s [summer internship](https://www.tidyverse.org/blog/2025/01/tidymodels-2025-internship/) focuses on supervised feature selection methods. We’ll also have releases (and probably another package) for these tools. These should come to fruition (and CRAN) before or around August 2025. Acknowledgements[](#acknowledgements) -------------------------------------- We want to sincerely thank everyone who contributed to these packages since their previous versions: [@AlbertoImg](https://github.com/AlbertoImg) , [@asb2111](https://github.com/asb2111) , [@balraadjsings](https://github.com/balraadjsings) , [@bcjaeger](https://github.com/bcjaeger) , [@beansrowning](https://github.com/beansrowning) , [@BrennanAntone](https://github.com/BrennanAntone) , [@cheryldietrich](https://github.com/cheryldietrich) , [@chillerb](https://github.com/chillerb) , [@conarr5](https://github.com/conarr5) , [@corybrunson](https://github.com/corybrunson) , [@dajmcdon](https://github.com/dajmcdon) , [@davidrsch](https://github.com/davidrsch) , [@Edgar-Zamora](https://github.com/Edgar-Zamora) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gaborcsardi](https://github.com/gaborcsardi) , [@gimholte](https://github.com/gimholte) , [@grantmcdermott](https://github.com/grantmcdermott) , [@grouptheory](https://github.com/grouptheory) , [@hfrick](https://github.com/hfrick) , [@ilaria-kode](https://github.com/ilaria-kode) , [@JamesHWade](https://github.com/JamesHWade) , [@jesusherranz](https://github.com/jesusherranz) , [@jkylearmstrong](https://github.com/jkylearmstrong) , [@joranE](https://github.com/joranE) , [@joscani](https://github.com/joscani) , [@Joscelinrocha](https://github.com/Joscelinrocha) , [@josho88](https://github.com/josho88) , [@joshuagi](https://github.com/joshuagi) , [@JosiahParry](https://github.com/JosiahParry) , [@jrosell](https://github.com/jrosell) , [@jrwinget](https://github.com/jrwinget) , [@KarlKoe](https://github.com/KarlKoe) , [@kscott-1](https://github.com/kscott-1) , [@lilykoff](https://github.com/lilykoff) , [@lionel-](https://github.com/lionel-) , [@LouisMPenrod](https://github.com/LouisMPenrod) , [@luisDVA](https://github.com/luisDVA) , [@marcelglueck](https://github.com/marcelglueck) , [@marcozanotti](https://github.com/marcozanotti) , [@martaalcalde](https://github.com/martaalcalde) , [@mattwarkentin](https://github.com/mattwarkentin) , [@mihem](https://github.com/mihem) , [@mitchellmanware](https://github.com/mitchellmanware) , [@naokiohno](https://github.com/naokiohno) , [@nhward](https://github.com/nhward) , [@npelikan](https://github.com/npelikan) , [@obgeneralao](https://github.com/obgeneralao) , [@owenjonesuob](https://github.com/owenjonesuob) , [@pbhogale](https://github.com/pbhogale) , [@Peter4801](https://github.com/Peter4801) , [@pgg1309](https://github.com/pgg1309) , [@reisner](https://github.com/reisner) , [@rfsaldanha](https://github.com/rfsaldanha) , [@rkb965](https://github.com/rkb965) , [@RobLBaker](https://github.com/RobLBaker) , [@RodDalBen](https://github.com/RodDalBen) , [@SantiagoD999](https://github.com/SantiagoD999) , [@shum461](https://github.com/shum461) , [@simonpcouch](https://github.com/simonpcouch) , [@szimmer](https://github.com/szimmer) , [@talegari](https://github.com/talegari) , [@therealjpetereit](https://github.com/therealjpetereit) , [@topepo](https://github.com/topepo) , [@walkerjameschris](https://github.com/walkerjameschris) , and [@ZWael](https://github.com/ZWael) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Categories [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Categories ---------- * [deep-dive](/categories/deep-dive/) (8) * [learn](/categories/learn/) (21) * [other](/categories/other/) (23) * [package](/categories/package/) (235) * [programming](/categories/programming/) (14) * [roundup](/categories/roundup/) (22) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Tags [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tags ---- * [agua](/tags/agua/) (1) * [ai](/tags/ai/) (1) * [baguette](/tags/baguette/) (1) * [Bayesian analysis](/tags/bayesian-analysis/) (1) * [bench](/tags/bench/) (1) * [bigrquery](/tags/bigrquery/) (5) * [bonsai](/tags/bonsai/) (2) * [book](/tags/book/) (2) * [broom](/tags/broom/) (1) * [brulee](/tags/brulee/) (1) * [bundle](/tags/bundle/) (1) * [butcher](/tags/butcher/) (1) * [callr](/tags/callr/) (2) * [censored](/tags/censored/) (4) * [cleancall](/tags/cleancall/) (1) * [clock](/tags/clock/) (1) * [conflicted](/tags/conflicted/) (1) * [correlation](/tags/correlation/) (1) * [corrr](/tags/corrr/) (2) * [cpp11](/tags/cpp11/) (1) * [databases](/tags/databases/) (5) * [dbplyr](/tags/dbplyr/) (11) * [desirability](/tags/desirability/) (1) * [devtools](/tags/devtools/) (13) * [dials](/tags/dials/) (4) * [discrim](/tags/discrim/) (2) * [dplyr](/tags/dplyr/) (34) * [dplyr-1-0-0](/tags/dplyr-1-0-0/) (9) * [dplyr-1-1-0](/tags/dplyr-1-1-0/) (6) * [dtplyr](/tags/dtplyr/) (2) * [ellmer](/tags/ellmer/) (1) * [embed](/tags/embed/) (3) * [embed-0-1-0](/tags/embed-0-1-0/) (1) * [finetune](/tags/finetune/) (2) * [forcats](/tags/forcats/) (4) * [fs](/tags/fs/) (2) * [gargle](/tags/gargle/) (5) * [generics](/tags/generics/) (1) * [ggplot2](/tags/ggplot2/) (22) * [ggplot2-3-5-0](/tags/ggplot2-3-5-0/) (4) * [GitHub Actions](/tags/github-actions/) (1) * [glue](/tags/glue/) (1) * [gmailr](/tags/gmailr/) (2) * [google](/tags/google/) (1) * [googledrive](/tags/googledrive/) (5) * [googlesheets4](/tags/googlesheets4/) (3) * [graphic-device](/tags/graphic-device/) (2) * [graphics](/tags/graphics/) (4) * [gt](/tags/gt/) (1) * [h2o](/tags/h2o/) (1) * [hardhat](/tags/hardhat/) (3) * [haven](/tags/haven/) (7) * [hiring](/tags/hiring/) (1) * [httr](/tags/httr/) (2) * [httr2](/tags/httr2/) (2) * [internship](/tags/internship/) (4) * [itdepends](/tags/itdepends/) (1) * [lifecycle](/tags/lifecycle/) (1) * [lintr](/tags/lintr/) (1) * [lobstr](/tags/lobstr/) (1) * [lubridate](/tags/lubridate/) (1) * [magrittr](/tags/magrittr/) (1) * [marquee](/tags/marquee/) (2) * [model](/tags/model/) (1) * [modeldata](/tags/modeldata/) (1) * [multilevelmod](/tags/multilevelmod/) (1) * [odbc](/tags/odbc/) (1) * [off-label](/tags/off-label/) (1) * [optimization](/tags/optimization/) (1) * [orbital](/tags/orbital/) (1) * [package](/tags/package/) (3) * [package maintenance](/tags/package-maintenance/) (2) * [parallelism](/tags/parallelism/) (2) * [parquet](/tags/parquet/) (2) * [parsnip](/tags/parsnip/) (23) * [patchwork](/tags/patchwork/) (1) * [pillar](/tags/pillar/) (2) * [pkgdown](/tags/pkgdown/) (8) * [plots](/tags/plots/) (1) * [plsmod](/tags/plsmod/) (1) * [poissonreg](/tags/poissonreg/) (1) * [postprocessing](/tags/postprocessing/) (1) * [probably](/tags/probably/) (1) * [processx](/tags/processx/) (1) * [purrr](/tags/purrr/) (4) * [quarto](/tags/quarto/) (1) * [r-lib](/tags/r-lib/) (49) * [ragg](/tags/ragg/) (5) * [readr](/tags/readr/) (3) * [readxl](/tags/readxl/) (3) * [recipes](/tags/recipes/) (16) * [reprex](/tags/reprex/) (1) * [rlang](/tags/rlang/) (4) * [roxygen2](/tags/roxygen2/) (5) * [rsample](/tags/rsample/) (8) * [rules](/tags/rules/) (1) * [rvest](/tags/rvest/) (1) * [s7](/tags/s7/) (1) * [scales](/tags/scales/) (4) * [shiny](/tags/shiny/) (1) * [shinylive](/tags/shinylive/) (1) * [shinymodels](/tags/shinymodels/) (1) * [slider](/tags/slider/) (1) * [spatialsample](/tags/spatialsample/) (1) * [spring cleaning](/tags/spring-cleaning/) (1) * [stacks](/tags/stacks/) (1) * [stringr](/tags/stringr/) (3) * [styler](/tags/styler/) (1) * [survey](/tags/survey/) (3) * [svglite](/tags/svglite/) (2) * [systemfonts](/tags/systemfonts/) (3) * [teaching](/tags/teaching/) (2) * [testthat](/tags/testthat/) (10) * [textrecipes](/tags/textrecipes/) (2) * [textshaping](/tags/textshaping/) (1) * [themis](/tags/themis/) (1) * [tibble](/tags/tibble/) (7) * [tidyclust](/tags/tidyclust/) (2) * [tidymodels](/tags/tidymodels/) (74) * [tidyposterior](/tags/tidyposterior/) (2) * [tidypredict](/tags/tidypredict/) (1) * [tidyr](/tags/tidyr/) (7) * [tidyselect](/tags/tidyselect/) (1) * [tidyverse](/tags/tidyverse/) (55) * [tidyverse-dev-day](/tags/tidyverse-dev-day/) (7) * [tune](/tags/tune/) (13) * [usethis](/tags/usethis/) (7) * [vctrs](/tags/vctrs/) (1) * [vdiffr](/tags/vdiffr/) (1) * [vetiver](/tags/vetiver/) (1) * [vroom](/tags/vroom/) (2) * [waldo](/tags/waldo/) (2) * [wasm](/tags/wasm/) (5) * [web](/tags/web/) (1) * [webassembly](/tags/webassembly/) (4) * [webr](/tags/webr/) (6) * [withr](/tags/withr/) (2) * [workflows](/tags/workflows/) (7) * [workflowsets](/tags/workflowsets/) (7) * [yardstick](/tags/yardstick/) (6) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidymodels [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidymodels [![](/blog/2025/02/tidymodels-2025-q1/thumbnail-sq.jpg)](/blog/2025/02/tidymodels-2025-q1/) [Q1 2025 tidymodels digest](/blog/2025/02/tidymodels-2025-q1/) [roundup](/categories/roundup) Max Kuhn A summary of the goings on for the tidymodels group in later 2024 and early 2025. [Read more ...](/blog/2025/02/tidymodels-2025-q1/) 2025/02/27 [![](/blog/2025/01/orbital-0-3-0/thumbnail-sq.jpg)](/blog/2025/01/orbital-0-3-0/) [orbital 0.3.0](/blog/2025/01/orbital-0-3-0/) [package](/categories/package) Emil Hvitfeldt orbital 0.3.0 is on CRAN! orbital now has classification support. [Read more ...](/blog/2025/01/orbital-0-3-0/) 2025/01/13 [![](/blog/2025/01/tidymodels-2025-internship/thumbnail-sq.jpg)](/blog/2025/01/tidymodels-2025-internship/) [tidymodels Internship for 2025](/blog/2025/01/tidymodels-2025-internship/) [other](/categories/other) Max Kuhn The tidymodels team is sponsoring a summer internship in 2025. [Read more ...](/blog/2025/01/tidymodels-2025-internship/) 2025/01/08 [![](/blog/2024/10/postprocessing-preview/thumbnail-sq.jpg)](/blog/2024/10/postprocessing-preview/) [Postprocessing is coming to tidymodels](/blog/2024/10/postprocessing-preview/) [roundup](/categories/roundup) Simon Couch, Hannah Frick, and Max Kuhn The tidymodels team has been hard at work on postprocessing, a set of features to adjust model predictions. The functionality includes a new package as well as changes across the framework. [Read more ...](/blog/2024/10/postprocessing-preview/) 2024/10/08 [![](/blog/2024/07/recipes-1-1-0/thumbnail-sq.jpg)](/blog/2024/07/recipes-1-1-0/) [recipes 1.1.0](/blog/2024/07/recipes-1-1-0/) [package](/categories/package) Emil Hvitfeldt recipes 1.1.0 is on CRAN! recipes now have better input checking and quality of life errors. [Read more ...](/blog/2024/07/recipes-1-1-0/) 2024/07/08 [![](/blog/2024/06/bonsai-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/bonsai-0-3-0/) [bonsai 0.3.0](/blog/2024/06/bonsai-0-3-0/) [package](/categories/package) Simon Couch A new release of the parsnip extension package bonsai introduces support for oblique random forests for classification and regression to tidymodels. [Read more ...](/blog/2024/06/bonsai-0-3-0/) 2024/06/25 [![](/blog/2024/04/tidymodels-2024-q1/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-2024-q1/) [Q1 2024 tidymodels digest](/blog/2024/04/tidymodels-2024-q1/) [roundup](/categories/roundup) Hannah Frick The tidymodels team has been busy working on all sorts of new features across the framework. [Read more ...](/blog/2024/04/tidymodels-2024-q1/) 2024/04/24 [![](/blog/2024/04/tune-1-2-0/thumbnail-sq.jpg)](/blog/2024/04/tune-1-2-0/) [tune 1.2.0](/blog/2024/04/tune-1-2-0/) [package](/categories/package) Simon Couch While we’ve written about survival analysis and machine learning fairness already, the newest tune release includes a number of other major changes. [Read more ...](/blog/2024/04/tune-1-2-0/) 2024/04/18 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2024/03/tidymodels-fairness/thumbnail-sq.jpg)](/blog/2024/03/tidymodels-fairness/) [Fair machine learning with tidymodels](/blog/2024/03/tidymodels-fairness/) [learn](/categories/learn) Simon Couch Recent tidymodels releases integrated a set of tools for assessing whether machine learning models treat groups of people differently. [Read more ...](/blog/2024/03/tidymodels-fairness/) 2024/03/21 * [««](/tags/tidymodels/) * « * [1](/tags/tidymodels/) * [2](/tags/tidymodels/page/2/) * [3](/tags/tidymodels/page/3/) *  …  * [8](/tags/tidymodels/page/8/) * [»](/tags/tidymodels/page/2/) * [»»](/tags/tidymodels/page/8/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # roundup [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Roundup [![](/blog/2025/02/tidymodels-2025-q1/thumbnail-sq.jpg)](/blog/2025/02/tidymodels-2025-q1/) [Q1 2025 tidymodels digest](/blog/2025/02/tidymodels-2025-q1/) [roundup](/categories/roundup) Max Kuhn A summary of the goings on for the tidymodels group in later 2024 and early 2025. [Read more ...](/blog/2025/02/tidymodels-2025-q1/) 2025/02/27 [![](/blog/2025/01/experiments-llm/thumbnail-sq.jpg)](/blog/2025/01/experiments-llm/) [Three experiments in LLM code assist with RStudio and Positron](/blog/2025/01/experiments-llm/) [roundup](/categories/roundup) Simon Couch We’ve been experimenting with LLM-powered tools to streamline R data science and package development. [Read more ...](/blog/2025/01/experiments-llm/) 2025/01/29 [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 [![](/blog/2024/10/quarto-live-0-1-1/thumbnail-sq.jpg)](/blog/2024/10/quarto-live-0-1-1/) [WebAssembly roundup part 3: Quarto Live 0.1.1](/blog/2024/10/quarto-live-0-1-1/) [roundup](/categories/roundup) George Stagg Quarto Live is a new Quarto extension that uses WebAssembly to bring interactive examples and dynamic code exercises to documents. Create highly engaging interactive elements for readers using standard Quarto markdown syntax. [Read more ...](/blog/2024/10/quarto-live-0-1-1/) 2024/10/15 [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 [![](/blog/2024/10/webr-0-4-2/thumbnail-sq.jpg)](/blog/2024/10/webr-0-4-2/) [WebAssembly roundup part 1: webR 0.4.2](/blog/2024/10/webr-0-4-2/) [roundup](/categories/roundup) George Stagg First in a series of blog posts about what’s new in R for WebAssembly. The latest release of webR 0.4.2 is now available, with updates to the viewer mechanism, support for displaying htmlwidgets, an IndexedDB virtual filesystem driver, and improved packaging for WebAssembly R binaries. [Read more ...](/blog/2024/10/webr-0-4-2/) 2024/10/11 [![](/blog/2024/10/postprocessing-preview/thumbnail-sq.jpg)](/blog/2024/10/postprocessing-preview/) [Postprocessing is coming to tidymodels](/blog/2024/10/postprocessing-preview/) [roundup](/categories/roundup) Simon Couch, Hannah Frick, and Max Kuhn The tidymodels team has been hard at work on postprocessing, a set of features to adjust model predictions. The functionality includes a new package as well as changes across the framework. [Read more ...](/blog/2024/10/postprocessing-preview/) 2024/10/08 [![](/blog/2024/04/tidymodels-2024-q1/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-2024-q1/) [Q1 2024 tidymodels digest](/blog/2024/04/tidymodels-2024-q1/) [roundup](/categories/roundup) Hannah Frick The tidymodels team has been busy working on all sorts of new features across the framework. [Read more ...](/blog/2024/04/tidymodels-2024-q1/) 2024/04/24 [![](/blog/2024/01/tidymodels-2023-q4/thumbnail-sq.jpg)](/blog/2024/01/tidymodels-2023-q4/) [Q4 2023 tidymodels digest](/blog/2024/01/tidymodels-2023-q4/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2024/01/tidymodels-2023-q4/) 2024/01/09 [![](/blog/2023/10/tidymodels-2023-q3/thumbnail-sq.jpg)](/blog/2023/10/tidymodels-2023-q3/) [Q3 2023 tidymodels digest](/blog/2023/10/tidymodels-2023-q3/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/10/tidymodels-2023-q3/) 2023/10/05 * [««](/categories/roundup/) * « * [1](/categories/roundup/) * [2](/categories/roundup/page/2/) * [3](/categories/roundup/page/3/) * [»](/categories/roundup/page/2/) * [»»](/categories/roundup/page/3/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Tidyverse [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) [![dplyr hex sticker](/css/images/hex/dplyr.png)](https://dplyr.tidyverse.org) [![ggplot2 hex sticker](/css/images/hex/ggplot2.png)](https://ggplot2.tidyverse.org) [![stringr hex sticker](/css/images/hex/stringr.png)](https://stringr.tidyverse.org) [![tibble hex sticker](/css/images/hex/tibble.png)](https://tibble.tidyverse.org) [![readr hex sticker](/css/images/hex/readr.png)](https://readr.tidyverse.org) [![forcats hex sticker](/css/images/hex/forcats.png)](https://forcats.tidyverse.org) [![tidyr hex sticker](/css/images/hex/tidyr.png)](https://tidyr.tidyverse.org) [![lubridate hex sticker](/css/images/hex/lubridate.png)](https://lubridate.tidyverse.org) [![purrr hex sticker](/css/images/hex/purrr.png)](https://purrr.tidyverse.org) R packages for data science The tidyverse is an opinionated [collection of R packages](/packages) designed for data science. All packages share an underlying design philosophy, grammar, and data structures. Install the complete tidyverse with: install.packages("tidyverse") Learn the tidyverse See how the tidyverse makes data science faster, easier and more fun with “R for Data Science (2e)". Read it [online](https://r4ds.hadley.nz/) , buy [the book](http://amzn.to/2aHLAQ1) or try another [resource](/learn/) from the community. [![R for Data Science book cover](/images/cover.png)](http://amzn.to/2aHLAQ1) ![](images/bee1.jpg) Need help? First learn how to make a [reprex](/help/#reprex) then [share it](/help/#where-to-ask) with others. The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Air, an extremely fast R formatter [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Air, an extremely fast R formatter ================================== ![](/blog/2025/02/air/thumbnail-wd.jpg) Photo by [Taylor Van Riper](https://unsplash.com/photos/above-cloud-photo-of-blue-skies-yQorCngxzwI)   2025/02/21   Davis Vaughan and Lionel Henry We’re thrilled to announce [Air](https://posit-dev.github.io/air/) , an extremely fast R formatter. Formatters are used to automatically style code, but I find that it’s much easier to show what Air can do rather than tell, so we’ll start with a few examples. In the video below, we’re inside [Positron](https://positron.posit.co/) and we’re looking at some unformatted code. Saving the file (yep, that’s it!) invokes Air, which automatically and instantaneously formats the code. Next, let’s go over to [RStudio](https://posit.co/products/open-source/rstudio/) . Here we’ve got a pipe chain that could use a little formatting. Like in Positron, just save the file: Lastly, we’ll jump back into Positron. Rather than formatting a single file on save, you might want to instead format an entire project (particularly when first adopting Air). To do so, just run `air format .` in a terminal from the project root, and Air will recursively format any R files it finds along the way (smartly excluding known generated files, like `cpp11.R`). Here we’ll run Air on dplyr for the first time ever, analyzing and reformatting over 150 files instantly: Within the tidyverse, we’re already using Air in some of our largest packages, like [dplyr](https://github.com/tidyverse/dplyr/pull/7662) , [tidyr](https://github.com/tidyverse/tidyr/pull/1591) , and [recipes](https://github.com/tidymodels/recipes/pull/1425) . Throughout the rest of this post you’ll learn what a formatter is, why you’d want to use one, and you’ll learn a little about how Air decides to format your R code. Note that Air is still in beta, so there may be some breaking changes over the next few releases. We also encourage you to use it in combination with a version control system, like git, so that you can clearly see the changes Air makes. That said, we still feel very confident in the current state of Air, and can’t wait for you to try it! Installing Air[](#installing-air) ---------------------------------- If you already know how formatters work and want to jump straight in, follow one of the guides below: * For Positron, Air is [available](https://open-vsx.org/extension/posit/air-vscode) on OpenVSX as an Extension. Install it from the Extensions pane within Positron, then read our [Positron guide](https://posit-dev.github.io/air/editor-vscode.html) . * For VS Code, Air is [available](https://marketplace.visualstudio.com/items?itemName=Posit.air-vscode) on the VS Code Marketplace as an Extension. Install it from the Extensions pane within VS Code, then read our [VS Code guide](https://posit-dev.github.io/air/editor-vscode.html) . * For RStudio, Air can be set as an external formatter, but you’ll need to install the command line tool for Air first. Read our [RStudio guide](https://posit-dev.github.io/air/editor-rstudio.html) to get started. Note that this is an experimental feature on the RStudio side, so the exact setup may change a little until it is fully stabilized. * For command line users, Air binaries can be installed using our [standalone installer scripts](https://posit-dev.github.io/air/cli.html) . For both Positron and VS Code, the most important thing to enable after installing the extension is format on save for R. You can do that by adding these lines to your `settings.json` file: { "[r]": { "editor.formatOnSave": true } } To open your `settings.json` file, run one of the following from the Command Palette: * Run `Preferences: Open User Settings (JSON)` to modify global user settings. * Run `Preferences: Open Workspace Settings (JSON)` to modify project specific settings. You may want to use this instead of setting the user level setting if you drop in on multiple projects, but not all of them use Air. If you work on a project with collaborators, we recommend that you check in these project specific settings to your repository to ensure that every collaborator is using the same formatting settings. If your preferred editor isn’t listed here, but does support the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) , then it is likely that we can add support for Air there as well. If you have any questions or run into issues installing or using Air, feel free to open an [issue](https://github.com/posit-dev/air/issues) ! What’s a formatter?[](#whats-a-formatter) ------------------------------------------ A formatter is in charge of the _layout_ of your R code. Formatters do not change the meaning of code; instead they ensure that whitespace, newlines, and other punctuation conform to a set of rules and standards, such as: * Making sure your code is **indented** with the appropriate amount of leading whitespace. By default, Air uses 2 spaces for indentation. You will see this indentation in pipelines: data |> ggplot(aes(x, y)) + geom_point() As well as in function calls: list( foo = 1, bar = 2 ) * Preventing your code from overflowing a given **line width**. By default, Air uses a line width of 80 characters. It enforces this by splitting long lines of code over multiple lines. For instance, notice how long these expressions are, they would “overflow” past 80 characters: band_members |> select(name) |> full_join(band_instruments2, by = join_by(name == artist)) left_join <- function(x, y, by = NULL, copy = FALSE, suffix = c(".x", ".y"), ..., keep = NULL) { UseMethod("left_join") } Air reformats these expressions by switching them from a horizontal layout (called “flat”) to a vertical one (called “expanded”): band_members |> select(name) |> full_join(band_instruments2, by = join_by(name == artist)) left_join <- function( x, y, by = NULL, copy = FALSE, suffix = c(".x", ".y"), ..., keep = NULL ) { UseMethod("left_join") } * Standardizing the whitespace around code elements. Have you ever had difficulties deciphering very dense code? 1+2:3*(4/5) Air reformats this expression to: 1 + 2:3 * (4 / 5) How does a formatter improve your workflow?[](#how-does-a-formatter-improve-your-workflow) ------------------------------------------------------------------------------------------- By using a formatter it might seem like you’re giving up control over the layout of your code. And indeed you are! However, putting Air in charge of styling your code has substantial advantages. First, it automatically forces you to write legible code that is neither too wide nor too narrow, with proper breathing room around syntactic elements. Having a formatter as a companion significantly improves the process of writing code as you no longer have to think about style - the formatter does that for you! Second, it reduces friction when working in a team. By agreeing to use a formatter in a project, collaborators no longer have to discuss styling and layout issues. Code sent to you by a colleague will adhere to the standards that you’re used to. Code review no longer has to be about style nitpicks and can focus on the substance of the changes instead. How does Air decide how to format your code?[](#how-does-air-decide-how-to-format-your-code) --------------------------------------------------------------------------------------------- Air tries to strike a balance between enforcing rigid rules and allowing authors some control over the layout. Our main source of styling rules is the [Tidyverse style guide](https://style.tidyverse.org) , but we occasionally deviate from these. There is a trend among modern formatters of being _opinionated_. Air certainly fits this trend and provides very few [configuration options](https://posit-dev.github.io/air/configuration.html) , mostly: the indent style (spaces versus tabs), the indent width, and the line width. However, Air also puts code authors in charge of certain aspects of the layout through the notion of **persistent line breaks**. In general, Air is in control of deciding where to put vertical space (line breaks) in your code. For instance if you write: dictionary <- list(bob = "apple", jill = "juice") Air will figure out that this expression fits on a single line without exceeding the line width. It will discard the line break and reformat to: dictionary <- list(bob = "apple", jill = "juice") However there are very specific places at which you can insert a line break that Air perceives as persistent: * Before the very first argument in a function call. This: # Persistent line break after `(` and before `bob` dictionary <- list( bob = "apple", jill = "juice") gets formatted as: dictionary <- list( bob = "apple", jill = "juice" ) * Before the very first right-hand side expression in a pipeline. This: # Persistent line break after `|>` and before `select` data |> select(foo) |> filter(!bar) gets formatted as: data |> select(foo) |> filter(!bar) A persistent line break will never be removed by Air. But you can remove it manually. Taking the last example, if you join the first lines like this: # Removed persistent line break after `(` dictionary <- list(bob = "apple", jill = "juice" ) # Removed persistent line break after `|>` data |> select(foo) |> filter(!bar) Air will recognize that you’ve removed the persistent line break, and reformat as: dictionary <- list(bob = "apple", jill = "juice") data |> select(foo) |> filter(!bar) The goal of this feature is to strike a balance between being opinionated and recognizing that users often know when taking up more vertical space results in more readable output. How can I disable formatting?[](#how-can-i-disable-formatting) --------------------------------------------------------------- If you need to disable formatting for a single expression, you can use a `# fmt: skip` comment. This is particularly useful for manual alignment. # This skips formatting for `list()` and its arguments, retaining the manual alignment # fmt: skip list( dollar = "USA", yen = "Japan", yuan = "China" ) # This skips formatting for `tribble()` and its arguments # fmt: skip tribble( ~x, ~y, 1, 2, ) If there is a file that Air should skip altogether, you can use a `# fmt: skip file` comment at the very top of the file. To learn more about these features, see the [documentation](https://posit-dev.github.io/air/formatter.html#disabling-formatting) . How can I use Air?[](#how-can-i-use-air) ----------------------------------------- As we’ve touched on above, Air can be integrated into your IDE to format code on every save. We expect that this will be the most common way to invoke Air, but there are a few other ways to use Air that we think are pretty cool: * In IDEs: * Format on save * Format selection * At the command line: * Format entire projects with `air format .` * Set up a git precommit hook to invoke Air before committing * In CI: * Use a GitHub Action to check that each PR conforms to formatting standards with `air format . --check`[1](#fn:1) * Use a GitHub Action to automatically format each PR by pushing the results of `air format` as a commit We don’t have guides for all of these use cases yet, but the best place to stay up to date is the [Air website](https://posit-dev.github.io/air/) . How is this different from styler?[](#how-is-this-different-from-styler) ------------------------------------------------------------------------- Air would not exist without the preexisting work and dedication poured into [styler](https://github.com/r-lib/styler) . Created by [Lorenz Walthert](https://github.com/lorenzwalthert) and [Kirill Müller](https://github.com/krlmlr) , styler proved that the R community does care about how their code is formatted, and has been the primary implementation of the [tidyverse style guide](https://style.tidyverse.org/) for many years. We’ve spoken to Lorenz about Air, and we are all very excited about what Air can do for the future of formatting in R. Air is different from styler in a few key ways: * Air is much faster. So much so that it enables new ways of using a formatter that were somewhat painful before, like formatting on every save, or formatting entire projects on every pull request. * Air is less configurable. As mentioned above, Air provides very few [configuration options](https://posit-dev.github.io/air/configuration.html) . * Air respects a line width, with a default of 80 characters. * Air does not require R to run. Unlike styler, which is an R package, Air is written in Rust and is distributed as a pre-compiled binary for many platforms. This makes Air easy to use across IDEs or on CI with very little setup required. How fast is “extremely fast”?[](#how-fast-is-extremely-fast) ------------------------------------------------------------- Air is written in Rust using the formatting infrastructure provided by [Biome](https://github.com/biomejs/biome) [2](#fn:2) . This is also the same infrastructure that [Ruff](https://github.com/astral-sh/ruff) , the fast Python formatter, originally forked from. Both of those projects are admired for their performance, and Air is no exception. One goal for Air is for “format on save” to be imperceptibly fast, encouraging you to keep it on at all times. Benchmarking formatters is a bit hand wavy due to some having built in caching, so bear with me, but one way to proxy this performance is by formatting a large single file, for example the 800+ line [join.R](https://github.com/tidyverse/dplyr/blob/main/R/join.R) in dplyr. Formatting this takes[3](#fn:3) : * 0.01 seconds with Air * 1 second with styler (no cache) So, ~100x faster for Air! If you make a few changes in the file after the first round of formatting and run the formatter again, then you get something like: * 0.01 seconds with Air * 0.5 seconds with styler (with cache) Half a second for styler might not sound that bad (and indeed, for a formatter written in R it’s pretty good), but it’s slow enough that you’ll “feel” it if you try and invoke styler on every save. But 0.01 seconds? You’ll never even know its running! The differences get even more drastic if you format entire projects. Formatting the ~150 R files in dplyr takes[4](#fn:4) : * 0.3 seconds with Air * 100 seconds with styler Over 300x faster! Out of curiosity, we also ran Air over all ~900 R files in base R and it finished in under 2 seconds. Wrapping up[](#wrapping-up) ---------------------------- By contributing this formatter to the R community, our objective is threefold: * Vastly improve your enjoyment of writing well-styled R code by removing the chore of editing whitespace. * Reduce friction in collaborative projects by establishing a consistent style once and for all. * Improve the overall readability of R code for the community. We hope that Air will prove to be a valuable companion in your daily workflow! * * * 1. The Shiny team already has a [GitHub Action](https://github.com/rstudio/shiny-workflows/tree/main/format-r-code) to help with this. We will likely work on refining this and incorporating it more officially into an Air or r-lib repository. [↩︎](#fnref:1) 2. Biome is an open source project maintained by community members, please consider [sponsoring them](https://github.com/sponsors/biomejs#sponsors) ! [↩︎](#fnref:2) 3. These benchmarks were run with `air format R/join.R` and `styler::style_file("R/join.R")`. [↩︎](#fnref:3) 4. With `air format .` and [`styler::style_pkg()`](https://styler.r-lib.org/reference/style_pkg.html) [↩︎](#fnref:4) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Three experiments in LLM code assist with RStudio and Positron [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Three experiments in LLM code assist with RStudio and Positron ============================================================== ![](/blog/2025/01/experiments-llm/thumbnail-wd.jpg) Photo by [Dibya Jyoti Ghosh](https://unsplash.com/photos/AgxNjvE8KTE)   2025/01/29   [ai](/tags/ai/) , [ellmer](/tags/ellmer/)   Simon Couch The last few months, I’ve been exploring how AI/LLMs might make my time developing R packages and doing data science more productive. This post will describe three experimental R packages— [pal](https://simonpcouch.github.io/pal/) , [ensure](https://simonpcouch.github.io/ensure/) , and [gander](https://simonpcouch.github.io/gander/) —that came out of that exploration, and the core tools underlying them. Taken together, I’ve found that these packages allow me to automate many of the less interesting parts of my work, turning all sorts of 45-second tasks into 5-second ones. Excitement from folks in the community has been very encouraging so far, and I’m looking forward to getting each of these packages buttoned up and sent off to CRAN in the coming weeks! Background[](#background) -------------------------- Twice a year, the tidyverse team sets a week aside for “spring cleaning,” bringing all of our R packages up to snuff with the most current tooling and standardizing various bits of our development process. Some of these updates can happen by calling a single function, while others are much more involved. One of those more involved updates is updating erroring code, transitioning away from base R (e.g.  [`stop()`](https://rdrr.io/r/base/stop.html) ), rlang (e.g.  [`rlang::abort()`](https://rlang.r-lib.org/reference/abort.html) ), [glue](https://glue.tidyverse.org/) , and homegrown combinations of them. cli’s new syntax is easier to work with as a developer and more visually pleasing as a user. In some cases, transitioning is almost as simple as Finding + Replacing [`rlang::abort()`](https://rlang.r-lib.org/reference/abort.html) to [`cli::cli_abort()`](https://cli.r-lib.org/reference/cli_abort.html) : # before: rlang::abort("`save_pred` can only be used if the initial results saved predictions.") # after: cli::cli_abort("{.arg save_pred} can only be used if the initial results saved predictions.") In others, there’s a mess of ad-hoc pluralization, [`paste0()`](https://rdrr.io/r/base/paste.html) s, glue interpolations, and other assorted nonsense to sort through: # before: extra_grid_params <- glue::single_quote(extra_grid_params) extra_grid_params <- glue::glue_collapse(extra_grid_params, sep = ", ") msg <- glue::glue( "The provided `grid` has the following parameter columns that have ", "not been marked for tuning by `tune()`: {extra_grid_params}." ) rlang::abort(msg) # after: cli::cli_abort( "The provided {.arg grid} has parameter columns that have not been marked for tuning by {.fn tune}: {.val {extra_grid_params}}." ) Total pain, especially with thousands upon thousands of error messages thrown across the tidyverse, r-lib, and tidymodels organizations. The week before our most recent spring cleaning, I participated in an internal Posit LLM hackathon, where a small group of employees would familiarize with interfacing with LLMs via APIs and then set aside a day or two to build something to make their work easier. Heading into our spring cleaning and dreading the task of updating thousands of these calls, I decided to look into how effectively LLMs could help me convert this code. Thus was born [clipal](https://github.com/simonpcouch/clipal) [1](#fn:1) , a (now-superseded) R package that allows users to select erroring code, press a keyboard shortcut, wait a moment, and watch the updated code be inlined in to the selection. ![A screencast of an RStudio session with an R file open in the source editor. 9 lines of ad-hoc erroring code are selected and, after a brief pause, replace with one call to [`cli::cli_abort()`](https://cli.r-lib.org/reference/cli_abort.html).](figs/clipal.gif) clipal was a _huge_ boost for us in the most recent spring cleaning. Depending on the code being updated, these erroring calls used to take 30 seconds to a few minutes. With clipal, though, the model could usually get the updated code 80% or 90% of the way there in a couple seconds. Up to this point, irritated by autocomplete and frustrated by the friction of copying and pasting code and typing out the same bits of context into chats again and again, I had been relatively skeptical that LLMs could make me more productive. After using clipal for a week, though, I began to understand how seamlessly LLMs could automate the cumbersome and uninteresting parts of my work. clipal itself is now superseded by pal, a more general solution to the problem that clipal solved. I’ve also written two additional packages like pal that solve two other classes of pal-like problems using similar tools, ensure and gander. In this post, I’ll write a bit about how I’ve used a pair of tools in three experiments that have made me much more productive as an R developer. Prerequisites: ellmer and the RStudio API[](#prerequisites-ellmer-and-the-rstudio-api) --------------------------------------------------------------------------------------- While clipal is now superseded, the package that supersedes it and its other two descendants makes use of the same two tools: [ellmer](https://github.com/tidyverse/ellmer) and the [RStudio API](https://rstudio.github.io/rstudioapi/) . Last year, Hadley Wickham and Joe Cheng began work on ellmer, a package that aims to make it easy to use large language models in R. For folks that have tried to use LLM APIs through HTTP requests, or interfaced with existing tools that wrap them like langchain, ellmer is pretty incredible. R users can initialize a Chat object using a predictably named function: library(ellmer) # to use a model like GPT-4o or GPT-4o-mini from OpenAI: ch <- chat_openai() # ...or a locally hosted ollama model: ch <- chat_ollama() # ...or Claude's Sonnet model: ch <- chat_claude() Then calling the output’s `$chat()` method returns a character response: ch$chat("When was R created? Be brief.") #> R was created in 1993 by Ross Ihaka and Robert Gentleman at #> the University of Auckland, New Zealand. There’s a whole lot more to ellmer, but this functionality alone was enough to make clipal happen. I could allow users to choose a Chat from whatever provider they prefer to power the addin and ellmer would take care of all of the details underneath the hood. The other puzzle piece here was how to get that character vector directly into the file so that the user didn’t have to copy and paste code from a chat interface into their document. The RStudio IDE supplies an API to interface with various bits of the RStudio UI through R code via the rstudioapi package. Notably, through R code, the package can read what’s inside of the user’s active selection and also write character vectors into that range. clipal could thus: * When triggered, read what’s inside of the selection using rstudioapi. * Pass that selection contents to an LLM along with a system prompt describing how to convert R erroring code to use cli using ellmer. (If you’re curious, the current draft of that prompt is [here](https://github.com/simonpcouch/pal/blob/1cd81736ee11cfaea1fd2466025dffcbdb845c3c/inst/prompts/cli-replace.md) .) * When the response is returned, replace the contents of the selection with the response using cli. This approach of using ellmer and the rstudioapi has its ups and downs. As for the advantages: * Our [Positron IDE](https://positron.posit.co/) has “shims” of the RStudio API, so whatever works in RStudio will also work in Positron. This means that the same shortcuts can be mapped to the same tool in either IDE and it will just work without me, as the developer, having to do anything.[2](#fn:2) * Since these packages are written in R, they have access to your R environment. This is quite the differentiator compared to the more language-agnostic tools out there—these packages can see the data frames you have loaded, the columns and column types in them, etc. When working with other tools for LLM code-assist that don’t have this information, the friction of printing out variable information from my R environment and pasting it into whatever interface is so high that I don’t even ask LLMs for help with tasks they’re otherwise totally capable of. * Using ellmer under the hood means that, once R users have set up model connections with ellmer, they can use the same configuration with any of these packages with minimal additional effort. So, clipal and the packages that followed it support whatever model providers their users want to use—OpenAI, Claude, local ollama models, and so on. If you can use it with ellmer, you can use it with these packages. As for the disadvantages, there are all sorts of UI bummers about this approach. Above all, these packages write directly to your files. This is great in that it removes the need to copy and paste, and when the model’s response is spot on, it’s awesome. At the same time, if the model starts rambling in an `.R` file or you want to confirm some difference between your previous code and the new code, the fact that these packages just write right into your files can be a bit annoying. Many other inline LLM code-assist tools out there are based on diffs—they show you proposed changes and some UI element that allows you to accept them, reject them, or ask for revisions. This requires one more step between asking for an LLM to do something and the thing actually being done, but saves the pain of lots of undoing or manually retrieving what code used to look like to verify the model’s work. pal[](#pal) ------------ ![The package hex, a yellow blob happily holding a checklist amid a purple background.](https://github.com/simonpcouch/pal/blob/main/inst/figs/logo.png?raw=true) After using clipal during our spring cleaning, I approached another spring cleaning task for the week: updating testing code. testthat 3.0.0 was released in 2020, bringing with it numerous changes that were both huge quality of life improvements for package developers and also highly breaking changes. While some of the task of converting legacy unit testing code to testthat 3e is relatively straightforward, other components can be quite tedious. Could I do the same thing for updating to testthat 3e that I did for transitioning to cli? I sloppily threw together a sister package to clipal that would convert tests for errors to snapshot tests, disentangle nested expectations, and transition from deprecated functions like `⁠expect_known_*()`. ⁠(If you’re interested, the current prompt for that functionality is [here](https://github.com/simonpcouch/pal/blob/1cd81736ee11cfaea1fd2466025dffcbdb845c3c/inst/prompts/testthat-replace.md) .) That sister package was also a huge boost for me, but the package reused as-is almost every piece of code from clipal other than the prompt. Thus, I realized that the proper solution would provide all of this scaffolding to attach a prompt to a keyboard shortcut, but allow for an arbitrary set of prompts to help automate these wonky, cumbersome tasks. The next week, [pal](https://simonpcouch.github.io/pal/) was born. The pal package ships with three prompts centered on package development: the cli pal and testthat pal mentioned previously, as well as the roxygen pal, which drafts minimal roxygen documentation based on a function definition. Here’s what pal’s interface looks like now: ![Another RStudio screencast. This time, a 12-line function definition is iteratively revised as the user selects lines of code and selects an entry in a dropdown menu, after which a model streams new code in place. In addition to converting erroring code, the model also drafts roxygen documentation for a function.](figs/pal.gif) Users can add custom prompts for whatever tasks they please and they’ll be included in the searchable dropdown shown above. I’ve been super appreciative of all of the love the package has received already, and I’ll be sending the package out to CRAN in the coming weeks. ensure[](#ensure) ------------------ While deciding on the initial set of prompts that pal would include, I really wanted to include some sort of “write unit tests for this function” pal. To really address this problem, though, requires violating two of pal’s core assumptions: * _All of the context that you need is in the selection and the prompt._ In the case of writing unit tests, it’s actually pretty important to have other pieces of context. If a package provides some object type `potato`, in order to write tests for some function that takes `potato` as input, it’s likely very important to know how potatoes are created and the kinds of properties they have. pal’s sister package for writing unit tests, ensure, can thus “see” the rest of the file that you’re working on, as well as context from neighboring files like other `.R` source files, the corresponding test file, and package vignettes, to learn about how to interface with the function arguments being tested. * _The LLM’s response can prefix, replace, or suffix the active selection in the same file._ In the case of writing unit tests for R, the place that tests actually ought to go is in a corresponding test file in `tests/testthat/`. Via the RStudio API, ensure can open up the corresponding test file and write to it rather than the source file where it was triggered from.[3](#fn:3) ![Another RStudio screencast. This time, the user selects around 20 lines of code situated in an R package and, after pressing a key command, the addin opens a corresponding test file and begins streaming unit testing code into the file. After the model completes streaming, the user runs the testing code and all tests pass.](figs/ensure.gif) So far, I haven’t spent as much time with ensure as I have with pal or gander, but I’ll be revisiting the package and sending it off to CRAN in the coming weeks. gander[](#gander) ------------------ [![The package hex, a goose hanging out amid a green background.](https://github.com/simonpcouch/gander/blob/main/inst/figs/gander.png?raw=true)](https://simonpcouch.github.io/gander/) pal really excels at things you do all the time. Providing custom prompts with lots of details about code syntax and your taste means that models will often provide code that’s almost exactly what you’d write yourself. On its own, though, pal is incomplete as a toolkit for LLM code-assist. What about one-off requests that are specific to the environment that I’m working in or things I only do every once in a long while? It’s nice to have a much more general tool that functions much more like a chat interface. At the same time, working with usual chat interfaces is quite high-friction, so much so that you’ll likely spend more time pasting in context from your files and R environmet than you would if you had just written the code yourself. There are all sorts of language-agnostic interfaces (or language-specific but not for R or RStudio) tools out there implementing this. You type some request with your cursor near some code, and then, in the backend, the tool assembles a bunch of context that will help the model respond more effectively. This is super helpful for many software engineering contexts, where most all of the context you need can be found in the contents of files. Data science differs a bit from software engineering here, though, in that the state of your R environment is just as important (or more so) than the contents of your files. For example, the lines of your files may show that you reference some data frame called `stackoverflow`, but what will _really_ help a model write R code to interface with that data frame is “seeing” it: what columns are in it, and what are their types and distributions? gander is a chat interface that allows models to see the data you’re working with. ![Another RStudio screencast. A script called example.R is open in the editor with lines library(ggplot2), data(stackoverflow), and stackoverflow. After highlighting the last line, the user triggers the addin and ask to plot the data in plain language, at which point code to plot the data using ggplot2 is streamed into the source file that uses the correct column names and a minimal style. The user iteratively calls the addin to refine the output.](figs/gander.gif) Behind the scenes, gander combines your selection (or lack thereof), inputted request, file type and contents, and R environment to dynamically assemble prompts to best enable models to tailor their responses to your R session. I use gander several times every day to turn 45-second tasks into 5-second ones and have been super stoked with how well-received it’s been among R folks so far. Compared to pal and ensure, this package feels like a much more substantial lift for data scientists specifically (rather than package developers). In the coming weeks, I’ll sand down some of its rough edges and send it off to CRAN. What’s next?[](#whats-next) ---------------------------- For now, all of these packages only live on my GitHub profile. In the coming weeks, I plan to revisit each of them, squash a bunch of bugs, and send them off to CRAN. That said, these packages are very much experimental. The user interface of writing directly to users’ files very much limits how useful these tools can be, and I think that the kinds of improvements to interface I’m hoping for may only be possible via some backend other than the RStudio API. I’m looking forward to seeing what that could look like. * * * 1. Pronounced “c-l-i pal.” [↩︎](#fnref:1) 2. In reality, there are bugs and differences here and there, but the development effort to get these packages working in Positron was relatively minimal. [↩︎](#fnref:2) 3. This is one gap between the RStudio API and Positron’s shims for it. The Positron shims currently don’t allow for toggling between files, so ensure isn’t available in Positron. [↩︎](#fnref:3) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # programming [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Programming [![](/blog/2025/02/air/thumbnail-sq.jpg)](/blog/2025/02/air/) [Air, an extremely fast R formatter](/blog/2025/02/air/) [programming](/categories/programming) Davis Vaughan and Lionel Henry We are thrilled to announce Air, a new R formatter. [Read more ...](/blog/2025/02/air/) 2025/02/21 [![](/blog/2023/11/tidymodels-errors-q4/thumbnail-sq.jpg)](/blog/2023/11/tidymodels-errors-q4/) [Three ways errors are about to get better in tidymodels](/blog/2023/11/tidymodels-errors-q4/) [programming](/categories/programming) Simon Couch The tidymodels team’s biannual spring cleaning gave us a chance to revisit the way we raise some error messages. [Read more ...](/blog/2023/11/tidymodels-errors-q4/) 2023/11/10 [![](/blog/2023/04/performant-packages/thumbnail-sq.jpg)](/blog/2023/04/performant-packages/) [Writing performant code with tidy tools](/blog/2023/04/performant-packages/) [programming](/categories/programming) Simon Couch When performance becomes an issue for code using tidy interfaces, switching to the backend tools used by tidy developers can offer substantial speedups. [Read more ...](/blog/2023/04/performant-packages/) 2023/04/18 [![](/blog/2023/03/cran-checks-compiled-code/thumbnail-sq.jpg)](/blog/2023/03/cran-checks-compiled-code/) [New CRAN requirements for packages with C and C++](/blog/2023/03/cran-checks-compiled-code/) [learn](/categories/learn) [programming](/categories/programming) Andy Teucher A few recent changes in CRAN requirements for packages containing C or C++ code have caused package developers some headaches. This post outlines the issues and provides solutions the tidyverse team has used to address them. [Read more ...](/blog/2023/03/cran-checks-compiled-code/) 2023/03/30 [![](/blog/2022/06/actions-2-0-0/thumbnail-sq.jpg)](/blog/2022/06/actions-2-0-0/) [GitHub Actions for R developers, v2](/blog/2022/06/actions-2-0-0/) [programming](/categories/programming) Gábor Csárdi We have updated our GitHub Actions at `r-lib/actions`. Consider upgrading to the new `v2` version, for faster and more reliable GHA jobs. [Read more ...](/blog/2022/06/actions-2-0-0/) 2022/06/01 [![](/blog/2021/08/waldo-0-3-0/thumbnail-sq.jpg)](/blog/2021/08/waldo-0-3-0/) [waldo 0.3.0](/blog/2021/08/waldo-0-3-0/) [package](/categories/package) [programming](/categories/programming) Hadley Wickham waldo 0.3.0 improves the display of data frame differences, and gives the objects being compared the ability to control the detail of their comparisons. [Read more ...](/blog/2021/08/waldo-0-3-0/) 2021/08/24 [![](/blog/2021/06/off-label-uses-in-ggplot2/thumbnail-sq.jpg)](/blog/2021/06/off-label-uses-in-ggplot2/) [Off-label uses in ggplot2](/blog/2021/06/off-label-uses-in-ggplot2/) [programming](/categories/programming) Thomas Lin Pedersen How can ggplot2 break with no breaking changes? This short post goes into detail with how and why the latest ggplot2 release disrupted some users workflow. [Read more ...](/blog/2021/06/off-label-uses-in-ggplot2/) 2021/06/24 [![](/blog/2020/04/self-cleaning-test-fixtures/thumbnail-sq.jpg)](/blog/2020/04/self-cleaning-test-fixtures/) [Self-cleaning test fixtures](/blog/2020/04/self-cleaning-test-fixtures/) [programming](/categories/programming) [learn](/categories/learn) Jenny Bryan A wild romp through environments – namely, the environments associated with functions and tests. How to adopt a low-impact lifestyle. [Read more ...](/blog/2020/04/self-cleaning-test-fixtures/) 2020/04/27 [![](/blog/2020/04/tibble-3-0-0/thumbnail-sq.jpg)](/blog/2020/04/tibble-3-0-0/) [tibble 3.0.0](/blog/2020/04/tibble-3-0-0/) [package](/categories/package) [programming](/categories/programming) Kirill Müller tibble 3.0.0 is on CRAN now! Tibbles are a modern reimagining of the data frame, keeping what time has shown to be effective, and throwing out what is not, with nicer default output too! This article describes the latest major release and provides an outlook on further developments [Read more ...](/blog/2020/04/tibble-3-0-0/) 2020/04/09 [![](/blog/2019/09/callr-task-q/callr-task-q-sq.jpg)](/blog/2019/09/callr-task-q/) [Multi Process Task Queue in 100 Lines of R Code](/blog/2019/09/callr-task-q/) [programming](/categories/programming) [deep-dive](/categories/deep-dive) Gábor Csárdi We use `callr::r_session` to implement a worker pool and task queue in 100 lines of R code. [Read more ...](/blog/2019/09/callr-task-q/) 2019/09/09 * [««](/categories/programming/) * « * [1](/categories/programming/) * [2](/categories/programming/page/2/) * [»](/categories/programming/page/2/) * [»»](/categories/programming/page/2/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ellmer [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Ellmer [![](/blog/2025/01/experiments-llm/thumbnail-sq.jpg)](/blog/2025/01/experiments-llm/) [Three experiments in LLM code assist with RStudio and Positron](/blog/2025/01/experiments-llm/) [roundup](/categories/roundup) Simon Couch We’ve been experimenting with LLM-powered tools to streamline R data science and package development. [Read more ...](/blog/2025/01/experiments-llm/) 2025/01/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # nanoparquet 0.4.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) nanoparquet 0.4.0 ================= ![](/blog/2025/01/nanoparquet-0-4-0/thumbnail-wd.jpg) Photo by [Michael Foster](https://www.pexels.com/photo/person-running-in-the-hallway-796545/)   2025/01/28   [parquet](/tags/parquet/)   Gábor Csárdi We’re thrilled to announce the release of [nanoparquet](https://nanoparquet.r-lib.org/) 0.4.0. nanoparquet is an R package that reads and writes Parquet files. You can install it from CRAN with: install.packages("nanoparquet") This blog post will show the most important new features of nanoparquet 0.4.0: You can see a full list of changes in the [release notes](https://nanoparquet.r-lib.org/news/index.html#nanoparquet-040) . Brand new `read_parquet()`[](#brand-new-read_parquet) ------------------------------------------------------ nanoparquet 0.4.0 comes with a completely rewritten Parquet reader. The new version has an architecture that is easier to embed into R, and facilitates fantastic new features, like [`append_parquet()`](https://nanoparquet.r-lib.org/reference/append_parquet.html) and the new `col_select` argument. (More to come!) The new reader is also much faster, see the “Benchmarks” chapter. Read a subset of columns[](#read-a-subset-of-columns) ------------------------------------------------------ [`read_parquet()`](https://nanoparquet.r-lib.org/reference/read_parquet.html) now has a new argument called `col_select`, that lets you read a subset of the columns from the Parquet file. Unlike for row oriented file formats like CSV, this means that the reader never needs to touch the columns that are not needed for. The time required for reading a subset of columns is independent of how many more columns the Parquet file might have! You can either use column indices or column names to specify the columns to read. Here is an example. library(nanoparquet) library(pillar) This is the [`nycflights13::flights`](https://rdrr.io/pkg/nycflights13/man/flights.html) data set: read_parquet( "flights.parquet", col_select = c("dep_time", "arr_time", "carrier") ) #> # A data frame: 336,776 × 3 #> dep_time arr_time carrier #> #> 1 517 830 UA #> 2 533 850 UA #> 3 542 923 AA #> 4 544 1004 B6 #> 5 554 812 DL #> 6 554 740 UA #> 7 555 913 B6 #> 8 557 709 EV #> 9 557 838 B6 #> 10 558 753 AA #> # ℹ 336,766 more rows Use [`read_parquet_schema()`](https://nanoparquet.r-lib.org/reference/read_parquet_schema.html) if you want to see the structure of the Parquet file first: read_parquet_schema("flights.parquet") #> # A data frame: 20 × 12 #> file_name name r_type type type_length repetition_type converted_type #> #> 1 flights.parquet sche… NA NA NA NA NA #> 2 flights.parquet year integ… INT32 NA REQUIRED INT_32 #> 3 flights.parquet month integ… INT32 NA REQUIRED INT_32 #> 4 flights.parquet day integ… INT32 NA REQUIRED INT_32 #> 5 flights.parquet dep_… integ… INT32 NA OPTIONAL INT_32 #> 6 flights.parquet sche… integ… INT32 NA REQUIRED INT_32 #> 7 flights.parquet dep_… double DOUB… NA OPTIONAL NA #> 8 flights.parquet arr_… integ… INT32 NA OPTIONAL INT_32 #> 9 flights.parquet sche… integ… INT32 NA REQUIRED INT_32 #> 10 flights.parquet arr_… double DOUB… NA OPTIONAL NA #> 11 flights.parquet carr… chara… BYTE… NA REQUIRED UTF8 #> 12 flights.parquet flig… integ… INT32 NA REQUIRED INT_32 #> 13 flights.parquet tail… chara… BYTE… NA OPTIONAL UTF8 #> 14 flights.parquet orig… chara… BYTE… NA REQUIRED UTF8 #> 15 flights.parquet dest chara… BYTE… NA REQUIRED UTF8 #> 16 flights.parquet air_… double DOUB… NA OPTIONAL NA #> 17 flights.parquet dist… double DOUB… NA REQUIRED NA #> 18 flights.parquet hour double DOUB… NA REQUIRED NA #> 19 flights.parquet minu… double DOUB… NA REQUIRED NA #> 20 flights.parquet time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC… #> # ℹ 5 more variables: logical_type >, num_children , scale , #> # precision , field_id The output of [`read_parquet_schema()`](https://nanoparquet.r-lib.org/reference/read_parquet_schema.html) also shows you the R type that nanoparquet will use for each column. Appending to Parquet files[](#appending-to-parquet-files) ---------------------------------------------------------- The new [`append_parquet()`](https://nanoparquet.r-lib.org/reference/append_parquet.html) function makes it easy to append new data to a Parquet file, without first reading the whole file into memory. The schema of the file and the schema new data must match of course. Lets merge [`nycflights13::flights`](https://rdrr.io/pkg/nycflights13/man/flights.html) and [`nycflights23::flights`](https://moderndive.github.io/nycflights23/reference/flights.html) : file.copy("flights.parquet", "allflights.parquet", overwrite = TRUE) #> [1] TRUE append_parquet(nycflights23::flights, "allflights.parquet") [`read_parquet_info()`](https://nanoparquet.r-lib.org/reference/read_parquet_info.html) returns the most basic information about a Parquet file: read_parquet_info("flights.parquet") #> # A data frame: 1 × 7 #> file_name num_cols num_rows num_row_groups file_size parquet_version #> #> 1 flights.parquet 19 336776 1 5687737 1 #> # ℹ 1 more variable: created_by read_parquet_info("allflights.parquet") #> # A data frame: 1 × 7 #> file_name num_cols num_rows num_row_groups file_size parquet_version #> #> 1 allflights.parquet 19 772128 1 13490997 1 #> # ℹ 1 more variable: created_by Note that you should probably still create a backup copy of the original file when using [`append_parquet()`](https://nanoparquet.r-lib.org/reference/append_parquet.html) . If the appending process is interrupted by a power failure, then you might end up with an incomplete and invalid Parquet file. Schemas and type conversions[](#schemas-and-type-conversions) -------------------------------------------------------------- In nanoparquet 0.4.0 [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) takes a `schema` argument that can customize the R to Parquet type mappings. For example by default [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) writes an R character vector as a `STRING` Parquet type. If you’d like to write a certain character column as an `ENUM` type[1](#fn:1) instead, you’ll need to specify that in `schema`: write_parquet( nycflights13::flights, "newflights.parquet", schema = parquet_schema(carrier = "ENUM") ) read_parquet_schema("newflights.parquet") #> # A data frame: 20 × 12 #> file_name name r_type type type_length repetition_type converted_type #> #> 1 newflights.par… sche… NA NA NA NA NA #> 2 newflights.par… year integ… INT32 NA REQUIRED INT_32 #> 3 newflights.par… month integ… INT32 NA REQUIRED INT_32 #> 4 newflights.par… day integ… INT32 NA REQUIRED INT_32 #> 5 newflights.par… dep_… integ… INT32 NA OPTIONAL INT_32 #> 6 newflights.par… sche… integ… INT32 NA REQUIRED INT_32 #> 7 newflights.par… dep_… double DOUB… NA OPTIONAL NA #> 8 newflights.par… arr_… integ… INT32 NA OPTIONAL INT_32 #> 9 newflights.par… sche… integ… INT32 NA REQUIRED INT_32 #> 10 newflights.par… arr_… double DOUB… NA OPTIONAL NA #> 11 newflights.par… carr… chara… BYTE… NA REQUIRED ENUM #> 12 newflights.par… flig… integ… INT32 NA REQUIRED INT_32 #> 13 newflights.par… tail… chara… BYTE… NA OPTIONAL UTF8 #> 14 newflights.par… orig… chara… BYTE… NA REQUIRED UTF8 #> 15 newflights.par… dest chara… BYTE… NA REQUIRED UTF8 #> 16 newflights.par… air_… double DOUB… NA OPTIONAL NA #> 17 newflights.par… dist… double DOUB… NA REQUIRED NA #> 18 newflights.par… hour double DOUB… NA REQUIRED NA #> 19 newflights.par… minu… double DOUB… NA REQUIRED NA #> 20 newflights.par… time… POSIX… INT64 NA REQUIRED TIMESTAMP_MIC… #> # ℹ 5 more variables: logical_type >, num_children , scale , #> # precision , field_id Here we wrote the `carrier` column as `ENUM`, and left the other other columns to use the default type mappings. See the [`?nanoparquet-types`](https://nanoparquet.r-lib.org/reference/nanoparquet-types.html#r-s-data-types) manual page for the possible type mappings (lots of new ones!) and also for the default ones. Encodings[](#encodings) ------------------------ It is now also possible to customize the encoding of each column in [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) , via the `encoding` argument. By default [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) tries to choose a good encoding based on the type and the values of a column. E.g. it checks a small sample for repeated values to decide if it is worth using a dictionary encoding (`RLE_DICTIONARY`). If [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) gets it wrong, use the `encoding` argument to force an encoding. The following forces the `PLAIN` encoding for all columns. This encoding is very fast to write, but creates a larger file. You can also specify different encodings for different columns, see the [`write_parquet()` manual page](https://nanoparquet.r-lib.org/reference/write_parquet.html) . write_parquet( nycflights13::flights, "plainflights.parquet", encoding = "PLAIN" ) file.size("flights.parquet") #> [1] 5687737 file.size("plainflights.parquet") #> [1] 11954574 See more about the implemented encodings and how the defaults are selected in the [`parquet-encodings` manual page](https://nanoparquet.r-lib.org/reference/parquet-encodings.html) . API changes[](#api-changes) ---------------------------- Some nanoparquet functions have new, better names in nanoparquet 0.4.0. In particular, all functions that read from a Parquet file have a `read_parquet` prefix now. The old functions still work, with a warning. Also, the [`parquet_schema()`](https://nanoparquet.r-lib.org/reference/parquet_schema.html) function is now for creating a new Parquet schema from scratch, and not for inferring a schema from a data frame (use [`infer_parquet_schema()`](https://nanoparquet.r-lib.org/reference/infer_parquet_schema.html) ) or for reading the schema from a Parquet file (use [`read_parquet_schema()`](https://nanoparquet.r-lib.org/reference/read_parquet_schema.html) ). [`parquet_schema()`](https://nanoparquet.r-lib.org/reference/parquet_schema.html) falls back to the old behaviour when called with a file name, with a warning, so this is not a breaking change (yet), and old code still works. See the complete list of API changes in the [Changelog](https://nanoparquet.r-lib.org/news/index.html) . Benchmarks[](#benchmarks) -------------------------- We are very excited about the performance of the new Parquet reader, and the Parquet writer was always quite speedy, so we ran a simple benchmark. We compared nanoparquet to the Parquet implementations in Apache Arrow and DuckDB, and also to CSV readers and writers, on a real data set, for samples of 330k, 6.7 million and 67.4 million rows (40MB, 800MB and 8GB in memory). For these data nanoparquet is indeed very competitive with both Arrow and DuckDB. You can see the full results [on the website](https://nanoparquet.r-lib.org/articles/benchmarks.html) . Other changes[](#other-changes) -------------------------------- Other important changes in nanoparquet 0.4.0 include: * [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) can now write multiple row groups. By default it puts at most 10 million rows in a single row group. (This is subject to [https://nanoparquet.r-lib.org/references/parquet\_options.html](https://nanoparquet.r-lib.org/references/parquet_options.html) ) on how to change the default. * [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) now writes minimum and maximum statistics (by default) for most Parquet types. See the [`parquet_options()` manual page](https://nanoparquet.r-lib.org/reference/parquet_options.html) on how to turn this off, which will probably make the writer faster. * [`write_parquet()`](https://nanoparquet.r-lib.org/reference/write_parquet.html) can now write version 2 data pages. The default is still version 1, but it might change in the future. * New `compression_level` option to select the compression level manually. * [`read_parquet()`](https://nanoparquet.r-lib.org/reference/read_parquet.html) can now read from an R connection. Acknowledgements[](#acknowledgements) -------------------------------------- [@alvarocombo](https://github.com/alvarocombo) , [@D3SL](https://github.com/D3SL) , [@gaborcsardi](https://github.com/gaborcsardi) , and [@RealTYPICAL](https://github.com/RealTYPICAL) . * * * 1. A Parquet `ENUM` type is very similar to a factor in R. [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ai [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Ai [![](/blog/2025/01/experiments-llm/thumbnail-sq.jpg)](/blog/2025/01/experiments-llm/) [Three experiments in LLM code assist with RStudio and Positron](/blog/2025/01/experiments-llm/) [roundup](/categories/roundup) Simon Couch We’ve been experimenting with LLM-powered tools to streamline R data science and package development. [Read more ...](/blog/2025/01/experiments-llm/) 2025/01/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # httr2 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Httr2 [![](/blog/2025/01/httr2-1-1-0/thumbnail-sq.jpg)](/blog/2025/01/httr2-1-1-0/) [httr2 1.1.0](/blog/2025/01/httr2-1-1-0/) [package](/categories/package) Hadley Wickham httr2 1.1.0 introduces powerful new streaming capabilities with `req_perform_connection()`, as well as comprehensive URL manipulation tools, improved AWS support, and a bunch of bug fixes. [Read more ...](/blog/2025/01/httr2-1-1-0/) 2025/01/20 [![](/blog/2023/11/httr2-1-0-0/thumbnail-sq.jpg)](/blog/2023/11/httr2-1-0-0/) [httr2 1.0.0](/blog/2023/11/httr2-1-0-0/) [package](/categories/package) Hadley Wickham httr2 is the successor to httr, providing a pipeable interface to generate HTTP requests and handle the responses. It’s focussed on the needs of an R user wrapping a modern web API, but is flexible enough to handle just about any HTTP related task. [Read more ...](/blog/2023/11/httr2-1-0-0/) 2023/11/14 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # package [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Package [![](/blog/2025/01/nanoparquet-0-4-0/thumbnail-sq.jpg)](/blog/2025/01/nanoparquet-0-4-0/) [nanoparquet 0.4.0](/blog/2025/01/nanoparquet-0-4-0/) [package](/categories/package) Gábor Csárdi nanoparquet 0.4.0 comes with a new and much faster `read_parquet()`, configurable type mappings in `write_parquet()`, and a new `append_parquet()`. [Read more ...](/blog/2025/01/nanoparquet-0-4-0/) 2025/01/28 [![](/blog/2025/01/httr2-1-1-0/thumbnail-sq.jpg)](/blog/2025/01/httr2-1-1-0/) [httr2 1.1.0](/blog/2025/01/httr2-1-1-0/) [package](/categories/package) Hadley Wickham httr2 1.1.0 introduces powerful new streaming capabilities with `req_perform_connection()`, as well as comprehensive URL manipulation tools, improved AWS support, and a bunch of bug fixes. [Read more ...](/blog/2025/01/httr2-1-1-0/) 2025/01/20 [![](/blog/2025/01/orbital-0-3-0/thumbnail-sq.jpg)](/blog/2025/01/orbital-0-3-0/) [orbital 0.3.0](/blog/2025/01/orbital-0-3-0/) [package](/categories/package) Emil Hvitfeldt orbital 0.3.0 is on CRAN! orbital now has classification support. [Read more ...](/blog/2025/01/orbital-0-3-0/) 2025/01/13 [![](/blog/2024/11/s7-0-2-0/thumbnail-sq.jpg)](/blog/2024/11/s7-0-2-0/) [S7 0.2.0](/blog/2024/11/s7-0-2-0/) [package](/categories/package) Tomasz Kalinowski and Hadley Wickham S7 is a new package that simplifies object-oriented programming (OOP) in R. It combines the simplicity of S3 with the structure of S4 to create a clearer system that’s accessible to everyone. [Read more ...](/blog/2024/11/s7-0-2-0/) 2024/11/07 [![](/blog/2024/09/patchwork-1-3-0/thumbnail-sq.jpg)](/blog/2024/09/patchwork-1-3-0/) [patchwork 1.3.0](/blog/2024/09/patchwork-1-3-0/) [package](/categories/package) Thomas Lin Pedersen patchwork 1.3.0 has just been released bringing refinements to the `free()` function and full on support for gt tables [Read more ...](/blog/2024/09/patchwork-1-3-0/) 2024/09/13 [![](/blog/2024/07/pkgdown-2-1-0/thumbnail-sq.jpg)](/blog/2024/07/pkgdown-2-1-0/) [pkgdown 2.1.0](/blog/2024/07/pkgdown-2-1-0/) [package](/categories/package) Hadley Wickham pkgdown 2.1.0 includes two major new features: support for quarto vignettes and a “light switch” that lets the reader switch between light and dark mode. It also contains a bunch of other improvements to both the user and the developer experience. [Read more ...](/blog/2024/07/pkgdown-2-1-0/) 2024/07/08 [![](/blog/2024/07/recipes-1-1-0/thumbnail-sq.jpg)](/blog/2024/07/recipes-1-1-0/) [recipes 1.1.0](/blog/2024/07/recipes-1-1-0/) [package](/categories/package) Emil Hvitfeldt recipes 1.1.0 is on CRAN! recipes now have better input checking and quality of life errors. [Read more ...](/blog/2024/07/recipes-1-1-0/) 2024/07/08 [![](/blog/2024/06/bonsai-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/bonsai-0-3-0/) [bonsai 0.3.0](/blog/2024/06/bonsai-0-3-0/) [package](/categories/package) Simon Couch A new release of the parsnip extension package bonsai introduces support for oblique random forests for classification and regression to tidymodels. [Read more ...](/blog/2024/06/bonsai-0-3-0/) 2024/06/25 [![](/blog/2024/06/nanoparquet-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/nanoparquet-0-3-0/) [nanoparquet 0.3.0](/blog/2024/06/nanoparquet-0-3-0/) [package](/categories/package) Gábor Csárdi Nanoparquet is a new R package that can read and write (flat) Parquet files. This post covers its features and limitations. [Read more ...](/blog/2024/06/nanoparquet-0-3-0/) 2024/06/20 [![](/blog/2024/05/marquee-0-1-0/thumbnail-sq.jpg)](/blog/2024/05/marquee-0-1-0/) [marquee 0.1.0](/blog/2024/05/marquee-0-1-0/) [package](/categories/package) Thomas Lin Pedersen The initial release brings markdown awareness to grid and ggplot2 to allow for rich text formatting in R graphics. [Read more ...](/blog/2024/05/marquee-0-1-0/) 2024/05/29 * [««](/categories/package/) * « * [1](/categories/package/) * [2](/categories/package/page/2/) * [3](/categories/package/page/3/) *  …  * [24](/categories/package/page/24/) * [»](/categories/package/page/2/) * [»»](/categories/package/page/24/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # parquet [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Parquet [![](/blog/2025/01/nanoparquet-0-4-0/thumbnail-sq.jpg)](/blog/2025/01/nanoparquet-0-4-0/) [nanoparquet 0.4.0](/blog/2025/01/nanoparquet-0-4-0/) [package](/categories/package) Gábor Csárdi nanoparquet 0.4.0 comes with a new and much faster `read_parquet()`, configurable type mappings in `write_parquet()`, and a new `append_parquet()`. [Read more ...](/blog/2025/01/nanoparquet-0-4-0/) 2025/01/28 [![](/blog/2024/06/nanoparquet-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/nanoparquet-0-3-0/) [nanoparquet 0.3.0](/blog/2024/06/nanoparquet-0-3-0/) [package](/categories/package) Gábor Csárdi Nanoparquet is a new R package that can read and write (flat) Parquet files. This post covers its features and limitations. [Read more ...](/blog/2024/06/nanoparquet-0-3-0/) 2024/06/20 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # graphics [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Graphics [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 [![](/blog/2022/11/ggplot2-3-4-0/thumbnail-sq.jpg)](/blog/2022/11/ggplot2-3-4-0/) [ggplot2 3.4.0](/blog/2022/11/ggplot2-3-4-0/) [package](/categories/package) Thomas Lin Pedersen ggplot2 3.4.0 is now on CRAN. Read all about the (mostly internal) changes that make up this release. [Read more ...](/blog/2022/11/ggplot2-3-4-0/) 2022/11/07 [![](/blog/2022/02/new-graphic-features/thumbnail-sq.jpg)](/blog/2022/02/new-graphic-features/) [ragg, svglite, and the new graphics features](/blog/2022/02/new-graphic-features/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen The graphics engine in R is continually improving, and the ragg and svglite graphic devices follow along in supporting them. This blog post outlines what is now possible, and what might come in the future. [Read more ...](/blog/2022/02/new-graphic-features/) 2022/02/25 [![](/blog/2020/08/taking-control-of-plot-scaling/thumbnail-sq.jpg)](/blog/2020/08/taking-control-of-plot-scaling/) [Taking Control of Plot Scaling](/blog/2020/08/taking-control-of-plot-scaling/) [learn](/categories/learn) Thomas Lin Pedersen Learn how to control scaling of the content in your plot when you render plots to different sizes. [Read more ...](/blog/2020/08/taking-control-of-plot-scaling/) 2020/08/21 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # marquee [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Marquee [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 [![](/blog/2024/05/marquee-0-1-0/thumbnail-sq.jpg)](/blog/2024/05/marquee-0-1-0/) [marquee 0.1.0](/blog/2024/05/marquee-0-1-0/) [package](/categories/package) Thomas Lin Pedersen The initial release brings markdown awareness to grid and ggplot2 to allow for rich text formatting in R graphics. [Read more ...](/blog/2024/05/marquee-0-1-0/) 2024/05/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # httr2 1.1.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) httr2 1.1.0 =========== ![](/blog/2025/01/httr2-1-1-0/thumbnail-wd.jpg) Photo by [Jose Morales](https://unsplash.com/photos/person-holding-two-baseballs-3k_FcShH0jY)   2025/01/20   [httr2](/tags/httr2/)   Hadley Wickham We’re chuffed to announce the release of [httr2 1.1.0](https://httr2.r-lib.org) . httr2 (pronounced “hitter2”) is a comprehensive HTTP client that provides a modern, pipeable API for working with web APIs. It builds on top of [{curl}](https://jeroen.r-universe.dev/curl) to provide features like explicit request objects, built-in rate limiting & retry tooling, comprehensive OAuth support, and secure handling of secrets and credentials. In this blog post, we’ll dive into the new streaming interface built around [`req_perform_connection()`](https://httr2.r-lib.org/reference/req_perform_connection.html) , explore the new suite of URL manipulation tools, and highlight a few of the other biggest changes (including better support for AWS and enhancements to the caching system), and update you on the lifecycle changes. This blog post includes the most important enhacenments in versions 1.0.1 through 1.0.7, where we’ve been iterating on various features and fixing _numerous_ bugs. For a complete list of changes, you can check the [GitHub release notes](https://github.com/r-lib/httr2/releases) or the [NEWS file](https://httr2.r-lib.org/news/index.html) . Installation[](#installation) ------------------------------ Install httr2 from CRAN with: install.packages("httr2") Streaming data[](#streaming-data) ---------------------------------- The headline feature of this release is a better API for streaming responses, where the body is not available immediately but is streamed back over time. This is particularly important for interacting with LLMs, where it’s needed to make chat responses feel snappy. You can try it out in [ellmer](https://ellmer.tidyverse.org) , our new package for chatting with LLMs from a variety of providers. The most important new function is [`req_perform_connection()`](https://httr2.r-lib.org/reference/req_perform_connection.html) , which supersedes the older callback-based [`req_perform_stream()`](https://httr2.r-lib.org/reference/req_perform_stream.html) . Unlike its predecessor, [`req_perform_connection()`](https://httr2.r-lib.org/reference/req_perform_connection.html) returns a regular response object with a connection object for the body: library(httr2) req <- request(example_url()) |> req_template("/stream-bytes/:n", n = 10240) resp <- req_perform_connection(req) resp #> #> GET http://127.0.0.1:49283/stream-bytes/10240 #> Status: 200 OK #> Content-Type: application/octet-stream #> Body: Streaming connection Once you have a streaming connection you can repeatedly call a `resp_stream_*()` function to pull down data in chunks, using [`resp_stream_is_complete()`](https://httr2.r-lib.org/reference/resp_stream_raw.html) to figure out when to stop. while (!resp_stream_is_complete(resp)) { bytes <- resp_stream_raw(resp, kb = 2) cat("Downloaded ", length(bytes), " bytes\n", sep = "") } #> Downloaded 2048 bytes #> Downloaded 2048 bytes #> Downloaded 2048 bytes #> Downloaded 2048 bytes #> Downloaded 2048 bytes #> Downloaded 0 bytes As well as [`resp_stream_raw()`](https://httr2.r-lib.org/reference/resp_stream_raw.html) , which returns a raw vector, you can use [`resp_stream_lines()`](https://httr2.r-lib.org/reference/resp_stream_raw.html) to stream lines and [`resp_stream_sse()`](https://httr2.r-lib.org/reference/resp_stream_raw.html) to stream [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) . URL manipulation tools[](#url-manipulation-tools) -------------------------------------------------- Working with URLs got easier with three new functions: [`url_modify()`](https://httr2.r-lib.org/reference/url_modify.html) , [`url_modify_query()`](https://httr2.r-lib.org/reference/url_modify.html) , and [`url_modify_relative()`](https://httr2.r-lib.org/reference/url_modify.html) . You can see how they work in the examples below: # url_modify() modifies components of a URL url_modify("https://example.com", hostname = "github.com") #> [1] "https://github.com/" url_modify("https://example.com", scheme = "http") #> [1] "http://example.com/" url_modify("https://example.com", path = "abc", query = list(foo = "bar")) #> [1] "https://example.com/abc?foo=bar" # url_modify_query() lets you modify individual query parameters # modifying an existing parameter: url_modify_query("http://example.com?a=1&b=2", a = 10) #> [1] "http://example.com/?b=2&a=10" # delete a parameter: url_modify_query("http://example.com?a=1&b=2", b = NULL) #> [1] "http://example.com/?a=1" # add a new parameter: url_modify_query("http://example.com?a=1&b=2", c = 3) #> [1] "http://example.com/?a=1&b=2&c=3" # url_modify_relative() navigates to a relative URL url_modify_relative("https://example.com/a/b/c.html", "/d/e/f.html") #> [1] "https://example.com/d/e/f.html" url_modify_relative("https://example.com/a/b/c.html", "C.html") #> [1] "https://example.com/a/b/C.html" url_modify_relative("https://example.com/a/b/c.html", "../B.html") #> [1] "https://example.com/a/B.html" We also added [`req_url_relative()`](https://httr2.r-lib.org/reference/req_url.html) to make it easier to navigate to a relative URL for an existing request. Other improvements[](#other-improvements) ------------------------------------------ There are a handful of other improvements that are worth highlighting: * We’ve made it easier to talk to AWS web services with [`req_auth_aws_v4()`](https://httr2.r-lib.org/reference/req_auth_aws_v4.html) for signing requests and [`resp_stream_aws()`](https://httr2.r-lib.org/reference/resp_stream_raw.html) for streaming responses. Special thanks goes to the [lifion-aws-event-stream](https://github.com/lifion/lifion-aws-event-stream/) project for providing a clear reference implementation. * We’ve run-down a long list of bugs that made [`req_cache()`](https://httr2.r-lib.org/reference/req_cache.html) unreliable. This includes improving the handling of header-only changes, better cache pruning, and new debugging options. If you’re working with a web API that supports caching, we highly recommend that you try it out. The next release of { [gh](https://github.com/r-lib/gh) } will use a cache by default, and my use of the dev version suggests that it gives a pretty nice performance improvment. * [`is_online()`](https://httr2.r-lib.org/reference/is_online.html) provides an easy way to check internet connectivity. * [`req_perform_promise()`](https://httr2.r-lib.org/reference/req_perform_promise.html) allows you to execute requests in the background (thanks to [@gergness](https://github.com/gergness) ) using an efficient approach that waits on curl socket activity (thanks to [@shikokuchuo](https://github.com/shikokuchuo) ). Breaking changes[](#breaking-changes) -------------------------------------- As httr2 continues to mature, we’ve made some lifecycle changes: * [`req_perform_iterative()`](https://httr2.r-lib.org/reference/req_perform_iterative.html) is now stable and no longer experimental. * [`req_perform_stream()`](https://httr2.r-lib.org/reference/req_perform_stream.html) is superseded by [`req_perform_connection()`](https://httr2.r-lib.org/reference/req_perform_connection.html) , as mentioned above. * [`with_mock()`](https://httr2.r-lib.org/reference/with_mocked_responses.html) and [`local_mock()`](https://httr2.r-lib.org/reference/with_mocked_responses.html) are defunct and will be rmeoved in the next release. Use [`with_mocked_responses()`](https://httr2.r-lib.org/reference/with_mocked_responses.html) and [`local_mocked_responses()`](https://httr2.r-lib.org/reference/with_mocked_responses.html) instead. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 76 folks who filed issues, created PRs and generally helped to make httr2 better! [@Aariq](https://github.com/Aariq) , [@AGeographer](https://github.com/AGeographer) , [@amael-ls](https://github.com/amael-ls) , [@anishjoni](https://github.com/anishjoni) , [@asadow](https://github.com/asadow) , [@atheriel](https://github.com/atheriel) , [@awpsoras](https://github.com/awpsoras) , [@billsanto](https://github.com/billsanto) , [@bonushenricus](https://github.com/bonushenricus) , [@botan](https://github.com/botan) , [@burgerga](https://github.com/burgerga) , [@CareCT](https://github.com/CareCT) , [@cderv](https://github.com/cderv) , [@cole-brokamp](https://github.com/cole-brokamp) , [@covid19ec](https://github.com/covid19ec) , [@datapumpernickel](https://github.com/datapumpernickel) , [@denskh](https://github.com/denskh) , [@deschen1](https://github.com/deschen1) , [@DyfanJones](https://github.com/DyfanJones) , [@erydit](https://github.com/erydit) , [@exetico](https://github.com/exetico) , [@fh-mthomson](https://github.com/fh-mthomson) , [@frzambra](https://github.com/frzambra) , [@gergness](https://github.com/gergness) , [@GreenGrassBlueOcean](https://github.com/GreenGrassBlueOcean) , [@guslipkin](https://github.com/guslipkin) , [@hadley](https://github.com/hadley) , [@i2z1](https://github.com/i2z1) , [@isachng93](https://github.com/isachng93) , [@IshuaWang](https://github.com/IshuaWang) , [@JamesHWade](https://github.com/JamesHWade) , [@jameslairdsmith](https://github.com/jameslairdsmith) , [@JBGruber](https://github.com/JBGruber) , [@jcheng5](https://github.com/jcheng5) , [@jeroen](https://github.com/jeroen) , [@jimbrig](https://github.com/jimbrig) , [@jjesusfilho](https://github.com/jjesusfilho) , [@jl5000](https://github.com/jl5000) , [@jmuhlenkamp](https://github.com/jmuhlenkamp) , [@jonthegeek](https://github.com/jonthegeek) , [@JosiahParry](https://github.com/JosiahParry) , [@jwimberl](https://github.com/jwimberl) , [@krjaworski](https://github.com/krjaworski) , [@m-muecke](https://github.com/m-muecke) , [@maarten-vermeyen](https://github.com/maarten-vermeyen) , [@MarekGierlinski](https://github.com/MarekGierlinski) , [@maxsutton](https://github.com/maxsutton) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@mkoohafkan](https://github.com/mkoohafkan) , [@MSHelm](https://github.com/MSHelm) , [@mstei4176](https://github.com/mstei4176) , [@mthomas-ketchbrook](https://github.com/mthomas-ketchbrook) , [@NateNohling](https://github.com/NateNohling) , [@nick-youngblut](https://github.com/nick-youngblut) , [@pbulsink](https://github.com/pbulsink) , [@PietrH](https://github.com/PietrH) , [@pkautio](https://github.com/pkautio) , [@plietar](https://github.com/plietar) , [@pmlefeuvre-met](https://github.com/pmlefeuvre-met) , [@rkrug](https://github.com/rkrug) , [@romainfrancois](https://github.com/romainfrancois) , [@salim-b](https://github.com/salim-b) , [@shikokuchuo](https://github.com/shikokuchuo) , [@simplyalexander](https://github.com/simplyalexander) , [@sluga](https://github.com/sluga) , [@stefanedwards](https://github.com/stefanedwards) , [@steveputman](https://github.com/steveputman) , [@tebancr](https://github.com/tebancr) , [@thohan88](https://github.com/thohan88) , [@tony2015116](https://github.com/tony2015116) , [@toobiwankenobi](https://github.com/toobiwankenobi) , [@verhovsky](https://github.com/verhovsky) , [@walinchus](https://github.com/walinchus) , [@werkstattcodes](https://github.com/werkstattcodes) , and [@zacdav-db](https://github.com/zacdav-db) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # systemfonts [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Systemfonts [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 [![](/blog/2021/02/modern-text-features/modern-text-features-sq.jpg)](/blog/2021/02/modern-text-features/) [Modern Text Features in R](/blog/2021/02/modern-text-features/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen ragg has taken a major leap forward in text-rendering capabilities with the latest releases of systemfonts, textshaping, and ragg itself. This post will go into detail with what is now possible and how it compares to the build in devices. [Read more ...](/blog/2021/02/modern-text-features/) 2021/02/06 [![](/blog/2020/05/updates-to-ragg-and-systemfonts/updates-to-ragg-and-systemfonts-sq.jpg)](/blog/2020/05/updates-to-ragg-and-systemfonts/) [Updates to ragg and systemfonts](/blog/2020/05/updates-to-ragg-and-systemfonts/) [package](/categories/package) Thomas Lin Pedersen We have recently updated some of our low-level graphics packages. Read on about what is new in ragg and systemfonts [Read more ...](/blog/2020/05/updates-to-ragg-and-systemfonts/) 2020/05/15 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # textshaping [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Textshaping [![](/blog/2025/01/text-rendering-updates/thumbnail-sq.jpg)](/blog/2025/01/text-rendering-updates/) [Updates to Text Rendering in R Graphics](/blog/2025/01/text-rendering-updates/) [roundup](/categories/roundup) Thomas Lin Pedersen There has been a recent flurry of updates to packages involved in rendering text from R. This blog post will go through what this means for you as a user and developer. [Read more ...](/blog/2025/01/text-rendering-updates/) 2025/01/17 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # orbital [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Orbital [![](/blog/2025/01/orbital-0-3-0/thumbnail-sq.jpg)](/blog/2025/01/orbital-0-3-0/) [orbital 0.3.0](/blog/2025/01/orbital-0-3-0/) [package](/categories/package) Emil Hvitfeldt orbital 0.3.0 is on CRAN! orbital now has classification support. [Read more ...](/blog/2025/01/orbital-0-3-0/) 2025/01/13 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Updates to Text Rendering in R Graphics [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Updates to Text Rendering in R Graphics ======================================= ![](/blog/2025/01/text-rendering-updates/thumbnail-wd.jpg) Photo by [Tim Mossholder](https://unsplash.com/photos/a-sign-on-the-side-of-a-building-in-a-foreign-language-O5TLxKaOZUs)   2025/01/17   [graphics](/tags/graphics/) , [marquee](/tags/marquee/) , [systemfonts](/tags/systemfonts/) , [textshaping](/tags/textshaping/)   Thomas Lin Pedersen > text rendering is one of those disciplines where, if you think you finally got it right, you can be 100% certain that you didn't > > — Thomas Lin Pedersen ([@thomasp85.com](https://bsky.app/profile/did:plc:6cf4jj62ofgoswlxcqdvtcg5?ref_src=embed) > ) [January 10, 2025 at 10:44 AM](https://bsky.app/profile/did:plc:6cf4jj62ofgoswlxcqdvtcg5/post/3lfevs4qe5k2l?ref_src=embed) No reason to hide the fact: Text rendering is complicated! When I set out to improve the support for modern text rendering features in R all those years ago, I don’t think I truly appreciated that fact. And probably for the better, since I’m not sure I would have taken on the task had I known. Taking the quote above as a universal truth (it comes from a reputable source after all), I’m sure I’ll never be fully done, but recent work on the whole stack at least makes me worry less about the correctness. This post will go through the changes that span the [systemfonts](https://github.com/r-lib/systemfonts) , [textshaping](https://github.com/r-lib/textshaping) , and [marquee](https://marquee.r-lib.org) packages and let you now how you, as a user or developer, should take advantage of them. Working with non-installed fonts[](#working-with-non-installed-fonts) ---------------------------------------------------------------------- The genesis of the systemfonts package was a need to be able to tap into the operating systems font library, so that whatever was installed on the system, would be available to graphics devices (assuming those devices used systemfonts). The scope of the package has gradually increased, and one of the needs that has become obvious over time, is a way to work with fonts, that aren’t installed on the system (E.g. if you want to bundle a font with a package, or if you are deploying a Shiny app that uses a specific font for the graphics). Until now, the `register_font()` and `register_variant()` functions have been the only option for letting systemfonts know about fonts other than those installed on the system. However, both of these functions were designed to circumvent limitations in the R graphics system when it comes to font selection (e.g. no way to use a “thin” font variant as the only weight option in the graphics system is bold yes/no), and as such were clunky to use for introducing new fonts. With the new version of systemfonts we get a dedicated way to tell systemfonts “please consider these font files as equals to the installed ones”. The function is called `add_fonts()` and all you need to do is to pass in a vector of paths to font files and these will then be reachable by systemfonts. # Add fonts from specific files systemfonts::add_fonts(c("path/to/font1.ttf", "path/to/font2.ttf")) In addition to this function, systemfonts also comes with `scan_local_fonts()` that looks in `./fonts` and `~/fonts` and adds any fonts located there. The function is called when systemfonts is loaded meaning that you can immediately uses fonts saved in these directories. This is great for deploying projects because all you need to do is to include a `fonts` folder at the root of you project and these fonts will then always be available wherever you deploy your code. While it is nice to have good access to the font files on your computer, the files has to come from somewhere. Nowadays that _somewhere_ is usually [Google Fonts](https://fonts.google.com) or some other online font repository. systemfonts is now aware of a few of these repositories (Google Fonts and [Font Squirrel](https://www.fontsquirrel.com) for now), and can search and download from these (using `search_web_fonts()`, `get_from_google_fonts()`, and `get_from_font_squirrel()`). The downloaded fonts are automatically added using `add_fonts()` so they are immediately available, and by default they are placed in `~/fonts` so that they persist across R sessions and projects. # Search and download fonts systemfonts::get_from_font_squirrel("Quicksand") systemfonts::get_from_google_fonts("Rubik Moonrocks") But what if you don’t want to think too much about all these details and just want to ensure that some specific font is available when a piece of code is running? In that case `require_font()` got you covered. This function allows you to state a dependency on a font in a script. The function scans the available fonts on the system and, if it doesn’t find anything, proceeds to look for the font in the online repositories, downloading it if it finds it. If that also fails the function will either throw an error, or map the required font to a fallback of your choosing: library(systemfonts) require_font("Rubik Moonrocks") plot.new() text(0.5, 0.5, "Fancy Font", family = "Rubik Moonrocks", cex = 6) ![plot of chunk unnamed-chunk-3](figure/unnamed-chunk-3-1.png) Remember that all of these niceties only goes into effect if you use a graphics device that uses systemfonts. For now, that more or less means that you should use ragg (you should use ragg anyway so that is not a terrible requirement). Getting to the Glyphs[](#getting-to-the-glyphs) ------------------------------------------------ Most fonts these days are based on a vector outline. That means that they can be scaled smoothly to any size and doesn’t take up a lot of storage space. It also means that there are polygons inside the font files and that these can be extracted! This is now possible with systemfonts and the new `glyph_outline()` and `glyph_raster()` functions. # Get the location of the glyph inside the font moonrocks <- font_info("Rubik Moonrocks") G <- glyph_info("G", path = moonrocks$path, index = moonrocks$index) # Extract the outline of the glyph and plot it outline <- glyph_outline(G$index, moonrocks$path, moonrocks$index, size = 400) grid::grid.path( x = outline$x, y = outline$y + 20, # To raise the baseline a bit id = outline$contour, default.units = "bigpts", gp = grid::gpar(fill = "grey", col = "black", lwd = 4) ) ![plot of chunk unnamed-chunk-4](figure/unnamed-chunk-4-1.png) Extracting them as polygons means that we can do all sorts of weird stuff with them if we so pleases: # Skew the glyph making it italic grid::grid.path( x = outline$x + outline$y * 0.4, y = outline$y + 20, # To raise the baseline a bit id = outline$contour, default.units = "bigpts", gp = grid::gpar(fill = "grey", col = "black", lwd = 4) ) ![plot of chunk unnamed-chunk-5](figure/unnamed-chunk-5-1.png) (real italic glyphs are designed to look good skewed, not just skewed versions of the regular glyphs) Remember how I said “most fonts” in the beginning of this section. There are still fonts that do not provide an outline, the prime example being most emoji fonts. The glyphs in such fonts are encoded as multiple bitmaps at fixed sizes (Microsofts emoji font going a different way by encoding them as SVGs). Since we can’t get to the data as outlines we can instead extract it as a raster: emoji <- font_info("emoji") dancer <- glyph_info("💃", path = emoji$path, index = emoji$index) raster <- glyph_raster(dancer$index, emoji$path, emoji$index, size = 400) grid::grid.draw(glyph_raster_grob(raster[[1]], 0, 50)) ![plot of chunk unnamed-chunk-6](figure/unnamed-chunk-6-1.png) In the above we used the `glyph_raster_grob()` helper function to create a raster grob with the correct scaling of the resulting raster. Raster extraction is not only for bitmap encoded fonts since it is easy to go from an outline to a raster (but not the other way around). Freetype (which systemfonts uses) includes a very efficient scanline rasterizer (the same as used in ragg) and we can thus get a raster version of any font: raster2 <- glyph_raster(G$index, moonrocks$path, moonrocks$index, size = 400) grid::grid.draw(glyph_raster_grob(raster2[[1]], 0, 20)) ![plot of chunk unnamed-chunk-7](figure/unnamed-chunk-7-1.png) The Way the Text Flows[](#the-way-the-text-flows) -------------------------------------------------- The thing that provoked me to writing the quote in the beginning of this blog post, was my work on the textshaping package. This package is largely invisible to the user but together with systemfonts it is responsible for laying out strings of text correctly. It figures out the location of every glyph and finds alternative fonts if the selected one doesn’t contain the needed glyph. textshaping powers ragg as well as marquee, doing the heavy lifting of translating a string of text into glyphs and locations. Part of converting a string into glyphs and coordinates (a process known as text shaping) is to figure out which way the text flows and act accordingly. For many people left-to-right flow is the natural text direction, but this is merely a cultural bias and many scripts with a different flow exists (arabic and hebrew being the two most dominant right-to-left flowing scripts). So, part of shaping requires figuring out what script a specific character belongs to and what direction it flows. This is all fairly simple when a string internally agrees on the direction of flow, but can get much more complicated when scripts are embedded within other scripts that doesn’t have the same flow (not to mention scripts embedded even deeper). Combine all of this with soft wrapping of text inside an embedded script and you got the recipe for a headache. textshaping (through me) already made the claim that it fully supported bi-directional text but it turned out that I severely misjudged the complexity. Because of this, the shaping engine has been rewritten almost from scratch. Based on the starting quote I can’t quite claim that it now works 100% correctly but it does pass all 91.707 test cases for bidirectional text provided by the Unicode consortium so there’s that. Again, it is unlikely that you will come into contact with textshaping directly so you will mostly experience these improvements in the way text just appears more correct (to the extend that this was ever an issue for you). The place you are most likely to stumble upon these changes is marquee, which uses textshaping under the hood. Styling in marquee has been expanded to include a `text_direction` setting. It defaults to `"auto"` which mean “deduce it from the text you get”, but you can also set it to `"ltr"` or `"rtl"` to set the direction explicitly. Be aware that this setting doesn’t change how single glyphs flow so you cannot use it to e.g. write arabic in left-to-right flow. Instead it governs the paragraph-level direction and thus how bi-directional text should be assembled. It also governs to which side indentation happen and the placement of bullets in bullet lists. Often, leaving it on the default value will work fine. There are also two new values for the `align` setting. `"auto"` picks either `"left"` or `"right"` depending on the text direction, while `"justified"` picks either `"justified-left"` or `"justified-right"`. This makes it much easier to work natively with right-to-left text as everything just looks as it should. To top it off, `classic_style()` gains an `ltr` argument that controls whether the styling in general should cater to left-to-right or right-to-left text. It controls things such as the position of the grey bar in quotation blocks and the indentation of nested lists. library(marquee) # Create a style specific for rtl text rtl_style <- classic_style( text_direction = "rtl", # Forces bidi text to be assembled from right to left align = "auto", # Will convert itself to "right" ltr = FALSE # Will move bullet padding and bar along quote blocks to the right ) A marquee for Everyone[](#a-marquee-for-everyone) -------------------------------------------------- Speaking of marquee, the biggest obstacle it has put in front of its users is that it is build on very new features in R. The ability to write text by placing glyphs one at a time was only added in R 4.2 and not every graphics device supports it yet (worse still, the implementation in the default macOS quartz device caused the session to crash). Again, ragg is your friend, but the Cairo devices also has excellent support. Text rendering, however, should always work. It is quite frustrating for text to not show up when you expect it to. Because of this it has been a clear plan to expand the support for marquee somehow. With the new version of marquee this is finally a reality. How does it work? Well, remember when we talked about extracting glyph outlines and rasters? If marquee encounters a graphics device that doesn’t provide the necessary features it will take matters into its own hands, by extracting all the necessary polygons and bitmaps and plot them. It is certainly not faster than relying on the optimized routines of the graphics device and it can also lead to visual degradation at smaller font sizes. But it works - everywhere. To show it off, here is an svg created with svglite which doesn’t have the required new features: text <- "_Fancy_ {.red Font}📝" m_grob <- marquee_grob( text, classic_style( body_font = "Rubik Moonrocks", base_size = 72 ) ) s <- svglite::svgstring(width = 7, height = 1.5) grid::grid.draw(m_grob) invisible(dev.off()) s() If you inspect the SVG above you’ll see that rather than being made up of text elements it is a collection of path and image elements. Again, it is unlikely that many people will use marquee like this. It is much more likely that they will encounter it through ggplot2 in the form of `geom_marquee()` and `element_marquee()`. The takeaway, however, is the same - it is now safe to use marquee even when you don’t know which graphics device will be used to render the text with. What’s Next?[](#whats-next) ---------------------------- Circling back to the starting quote. I’m 100% certain I’m not done yet. I believe the next big push will be proper support for vertical text in textshaping (it currently only deals with horizontal text). I also have some plans to get marquee to automatically translate the numbers in ordered lists into their proper representation in the script that is being used, so that e.g. ‘3.’ will be shown as ‘.٣’ when used with Arabic text. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # orbital 0.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) orbital 0.3.0 ============= ![](/blog/2025/01/orbital-0-3-0/thumbnail-wd.jpg) Photo by [SpaceX](https://www.pexels.com/photo/aerial-view-earth-exploration-flying-60132/)   2025/01/13   [tidymodels](/tags/tidymodels/) , [orbital](/tags/orbital/)   Emil Hvitfeldt We’re thrilled to announce the release of [orbital](https://orbital.tidymodels.org/) 0.3.0. orbital lets you predict in databases using tidymodels workflows. You can install it from CRAN with: install.packages("orbital") This blog post will cover the highlights, which are classification support and the new augment method. You can see a full list of changes in the [release notes](https://orbital.tidymodels.org/news/index.html#orbital-030) . Classification support[](#classification-support) -------------------------------------------------- The biggest improvement in this version is that [`orbital()`](https://orbital.tidymodels.org/reference/orbital.html) now works for supported classification models. See [vignette](https://orbital.tidymodels.org/articles/supported-models.html#supported-models) for list of all supported models. Let’s start by fitting a classification model on the `penguins` data set, using {xgboost} as the engine. rec_spec <- recipe(species ~ ., data = penguins) |> step_unknown(all_nominal_predictors()) |> step_dummy(all_nominal_predictors()) |> step_impute_mean(all_numeric_predictors()) |> step_zv(all_predictors()) lr_spec <- boost_tree() |> set_mode("classification") |> set_engine("xgboost") wf_spec <- workflow(rec_spec, lr_spec) wf_fit <- fit(wf_spec, data = penguins) With this fitted workflow object, we can call [`orbital()`](https://orbital.tidymodels.org/reference/orbital.html) on it to create an orbital object. orbital_obj <- orbital(wf_fit) orbital_obj #> #> ── orbital Object ────────────────────────────────────────────────────────────── #> • island = dplyr::if_else(is.na(island), "unknown", island) #> • sex = dplyr::if_else(is.na(sex), "unknown", sex) #> • island_Dream = as.numeric(island == "Dream") #> • island_Torgersen = as.numeric(island == "Torgersen") #> • sex_male = as.numeric(sex == "male") #> • sex_unknown = as.numeric(sex == "unknown") #> • bill_length_mm = dplyr::if_else(is.na(bill_length_mm), 43.92193, bill_l ... #> • bill_depth_mm = dplyr::if_else(is.na(bill_depth_mm), 17.15117, bill_dep ... #> • flipper_length_mm = dplyr::if_else(is.na(flipper_length_mm), 201, flipp ... #> • body_mass_g = dplyr::if_else(is.na(body_mass_g), 4202, body_mass_g) #> • island_Dream = dplyr::if_else(is.na(island_Dream), 0.3604651, island_Dr ... #> • island_Torgersen = dplyr::if_else(is.na(island_Torgersen), 0.1511628, i ... #> • sex_male = dplyr::if_else(is.na(sex_male), 0.4883721, sex_male) #> • sex_unknown = dplyr::if_else(is.na(sex_unknown), 0.03197674, sex_unknow ... #> • Adelie = 0 + dplyr::case_when((bill_depth_mm < 15.1 | is.na(bill_depth_ ... #> • Chinstrap = 0 + dplyr::case_when((island_Dream < 0.5 | is.na(island_Dre ... #> • Gentoo = 0 + dplyr::case_when((bill_depth_mm < 15.95 | is.na(bill_depth ... #> • .pred_class = dplyr::case_when(Adelie > Chinstrap & Adelie > Gentoo ~ " ... #> ──────────────────────────────────────────────────────────────────────────────── #> 18 equations in total. This object contains all the information that is needed to produce predictions. Which we can produce with [`predict()`](https://rdrr.io/r/stats/predict.html) . predict(orbital_obj, penguins) #> # A tibble: 344 × 1 #> .pred_class #> #> 1 Adelie #> 2 Adelie #> 3 Adelie #> 4 Adelie #> 5 Adelie #> 6 Adelie #> 7 Adelie #> 8 Adelie #> 9 Adelie #> 10 Adelie #> # ℹ 334 more rows The main thing to note here is that the orbital package produces character vectors instead of factors. This is done as a unifying approach since many databases don’t have factor types. Speaking of databases, you can [`predict()`](https://rdrr.io/r/stats/predict.html) on an orbital object using tables from databases. Below we create an ephemeral in-memory RSQLite database. library(DBI) library(RSQLite) con_sqlite <- dbConnect(SQLite(), path = ":memory:") penguins_sqlite <- copy_to(con_sqlite, penguins, name = "penguins_table") And we can predict with it like normal. All the calculations are sent to the database for execution. predict(orbital_obj, penguins_sqlite) #> # Source: SQL [?? x 1] #> # Database: sqlite 3.47.1 [] #> .pred_class #> #> 1 Adelie #> 2 Adelie #> 3 Adelie #> 4 Adelie #> 5 Adelie #> 6 Adelie #> 7 Adelie #> 8 Adelie #> 9 Adelie #> 10 Adelie #> # ℹ more rows This works the same with [many types of databases](https://orbital.tidymodels.org/articles/databases.html) . Classification is different from regression in part because it comes with multiple prediction types. The above example showed the default which is hard classification. You can set the type of prediction you want with the `type` argument to `orbital`. For classification models, possible options are `"class"` and `"prob"`. orbital_obj_prob <- orbital(wf_fit, type = c("class", "prob")) orbital_obj_prob #> #> ── orbital Object ────────────────────────────────────────────────────────────── #> • island = dplyr::if_else(is.na(island), "unknown", island) #> • sex = dplyr::if_else(is.na(sex), "unknown", sex) #> • island_Dream = as.numeric(island == "Dream") #> • island_Torgersen = as.numeric(island == "Torgersen") #> • sex_male = as.numeric(sex == "male") #> • sex_unknown = as.numeric(sex == "unknown") #> • bill_length_mm = dplyr::if_else(is.na(bill_length_mm), 43.92193, bill_l ... #> • bill_depth_mm = dplyr::if_else(is.na(bill_depth_mm), 17.15117, bill_dep ... #> • flipper_length_mm = dplyr::if_else(is.na(flipper_length_mm), 201, flipp ... #> • body_mass_g = dplyr::if_else(is.na(body_mass_g), 4202, body_mass_g) #> • island_Dream = dplyr::if_else(is.na(island_Dream), 0.3604651, island_Dr ... #> • island_Torgersen = dplyr::if_else(is.na(island_Torgersen), 0.1511628, i ... #> • sex_male = dplyr::if_else(is.na(sex_male), 0.4883721, sex_male) #> • sex_unknown = dplyr::if_else(is.na(sex_unknown), 0.03197674, sex_unknow ... #> • Adelie = 0 + dplyr::case_when((bill_depth_mm < 15.1 | is.na(bill_depth_ ... #> • Chinstrap = 0 + dplyr::case_when((island_Dream < 0.5 | is.na(island_Dre ... #> • Gentoo = 0 + dplyr::case_when((bill_depth_mm < 15.95 | is.na(bill_depth ... #> • .pred_class = dplyr::case_when(Adelie > Chinstrap & Adelie > Gentoo ~ " ... #> • norm = exp(Adelie) + exp(Chinstrap) + exp(Gentoo) #> • .pred_Adelie = exp(Adelie) / norm #> • .pred_Chinstrap = exp(Chinstrap) / norm #> • .pred_Gentoo = exp(Gentoo) / norm #> ──────────────────────────────────────────────────────────────────────────────── #> 22 equations in total. Notice how we can select both `"class"` and `"prob"`. The predictions now include both hard and soft class predictions. predict(orbital_obj_prob, penguins) #> # A tibble: 344 × 4 #> .pred_class .pred_Adelie .pred_Chinstrap .pred_Gentoo #> #> 1 Adelie 0.989 0.00554 0.00560 #> 2 Adelie 0.989 0.00554 0.00560 #> 3 Adelie 0.989 0.00554 0.00560 #> 4 Adelie 0.709 0.0245 0.267 #> 5 Adelie 0.989 0.00554 0.00560 #> 6 Adelie 0.989 0.00554 0.00560 #> 7 Adelie 0.989 0.00554 0.00560 #> 8 Adelie 0.989 0.00554 0.00560 #> 9 Adelie 0.979 0.00549 0.0158 #> 10 Adelie 0.980 0.00559 0.0148 #> # ℹ 334 more rows That works equally well in databases. predict(orbital_obj_prob, penguins_sqlite) #> # Source: SQL [?? x 4] #> # Database: sqlite 3.47.1 [] #> .pred_class .pred_Adelie .pred_Chinstrap .pred_Gentoo #> #> 1 Adelie 0.989 0.00554 0.00560 #> 2 Adelie 0.989 0.00554 0.00560 #> 3 Adelie 0.989 0.00554 0.00560 #> 4 Adelie 0.709 0.0245 0.267 #> 5 Adelie 0.989 0.00554 0.00560 #> 6 Adelie 0.989 0.00554 0.00560 #> 7 Adelie 0.989 0.00554 0.00560 #> 8 Adelie 0.989 0.00554 0.00560 #> 9 Adelie 0.979 0.00549 0.0158 #> 10 Adelie 0.980 0.00559 0.0148 #> # ℹ more rows New augment method[](#new-augment-method) ------------------------------------------ The users of tidymodels have found the [`augment()`](https://generics.r-lib.org/reference/augment.html) function to be a handy tool. This function performs predictions and returns them alongside the original data set. This release adds [`augment()`](https://generics.r-lib.org/reference/augment.html) support for orbital objects. augment(orbital_obj, penguins) #> # A tibble: 344 × 8 #> .pred_class species island bill_length_mm bill_depth_mm flipper_length_mm #> #> 1 Adelie Adelie Torgersen 39.1 18.7 181 #> 2 Adelie Adelie Torgersen 39.5 17.4 186 #> 3 Adelie Adelie Torgersen 40.3 18 195 #> 4 Adelie Adelie Torgersen NA NA NA #> 5 Adelie Adelie Torgersen 36.7 19.3 193 #> 6 Adelie Adelie Torgersen 39.3 20.6 190 #> 7 Adelie Adelie Torgersen 38.9 17.8 181 #> 8 Adelie Adelie Torgersen 39.2 19.6 195 #> 9 Adelie Adelie Torgersen 34.1 18.1 193 #> 10 Adelie Adelie Torgersen 42 20.2 190 #> # ℹ 334 more rows #> # ℹ 2 more variables: body_mass_g , sex The function works for most databases, but for technical reasons doesn’t work with all. It has been confirmed to not work work in spark databases or arrow tables. augment(orbital_obj, penguins_sqlite) #> # Source: SQL [?? x 8] #> # Database: sqlite 3.47.1 [] #> .pred_class species island bill_length_mm bill_depth_mm flipper_length_mm #> #> 1 Adelie Adelie Torgersen 39.1 18.7 181 #> 2 Adelie Adelie Torgersen 39.5 17.4 186 #> 3 Adelie Adelie Torgersen 40.3 18 195 #> 4 Adelie Adelie Torgersen NA NA NA #> 5 Adelie Adelie Torgersen 36.7 19.3 193 #> 6 Adelie Adelie Torgersen 39.3 20.6 190 #> 7 Adelie Adelie Torgersen 38.9 17.8 181 #> 8 Adelie Adelie Torgersen 39.2 19.6 195 #> 9 Adelie Adelie Torgersen 34.1 18.1 193 #> 10 Adelie Adelie Torgersen 42 20.2 190 #> # ℹ more rows #> # ℹ 2 more variables: body_mass_g , sex Acknowledgements[](#acknowledgements) -------------------------------------- A big thank you to all the people who have contributed to orbital since the release of v0.3.0: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@joscani](https://github.com/joscani) , [@jrosell](https://github.com/jrosell) , [@npelikan](https://github.com/npelikan) , and [@szimmer](https://github.com/szimmer) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # internship [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Internship [![](/blog/2025/01/joining-ggplot2/thumbnail-sq.jpg)](/blog/2025/01/joining-ggplot2/) [Joining the ggplot2 team](/blog/2025/01/joining-ggplot2/) [other](/categories/other) Teun van den Brand I joined the ggplot2 team and would like to share the experience. [Read more ...](/blog/2025/01/joining-ggplot2/) 2025/01/09 [![](/blog/2025/01/tidymodels-2025-internship/thumbnail-sq.jpg)](/blog/2025/01/tidymodels-2025-internship/) [tidymodels Internship for 2025](/blog/2025/01/tidymodels-2025-internship/) [other](/categories/other) Max Kuhn The tidymodels team is sponsoring a summer internship in 2025. [Read more ...](/blog/2025/01/tidymodels-2025-internship/) 2025/01/08 [![](/blog/2023/08/data-trail/thumbnail-sq.jpg)](/blog/2023/08/data-trail/) [Solutions for R4DS, 2e with Data Trail](/blog/2023/08/data-trail/) [learn](/categories/learn) Jabir Ghaffar, Davon Person, Mine Çetinkaya-Rundel, Tracy Teal Jabir and Davon from Data Trail worked with us to create the solutions manual for R for Data Science, 2nd edition. This blog post summarizes their experience and shares their reflections from working on this project. [Read more ...](/blog/2023/08/data-trail/) 2023/08/02 [![](/blog/2021/09/updating-to-cpp11/thumbnail-sq.jpg)](/blog/2021/09/updating-to-cpp11/) [Pathway to success - updating your package to cpp11](/blog/2021/09/updating-to-cpp11/) [learn](/categories/learn) Shelby Bearrows The cpp11 summer intern reviews the process they used to convert readxl to using cpp11. [Read more ...](/blog/2021/09/updating-to-cpp11/) 2021/09/10 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Ggplot2 [![](/blog/2025/01/joining-ggplot2/thumbnail-sq.jpg)](/blog/2025/01/joining-ggplot2/) [Joining the ggplot2 team](/blog/2025/01/joining-ggplot2/) [other](/categories/other) Teun van den Brand I joined the ggplot2 team and would like to share the experience. [Read more ...](/blog/2025/01/joining-ggplot2/) 2025/01/09 [![](/blog/2024/09/patchwork-1-3-0/thumbnail-sq.jpg)](/blog/2024/09/patchwork-1-3-0/) [patchwork 1.3.0](/blog/2024/09/patchwork-1-3-0/) [package](/categories/package) Thomas Lin Pedersen patchwork 1.3.0 has just been released bringing refinements to the `free()` function and full on support for gt tables [Read more ...](/blog/2024/09/patchwork-1-3-0/) 2024/09/13 [![](/blog/2024/05/marquee-0-1-0/thumbnail-sq.jpg)](/blog/2024/05/marquee-0-1-0/) [marquee 0.1.0](/blog/2024/05/marquee-0-1-0/) [package](/categories/package) Thomas Lin Pedersen The initial release brings markdown awareness to grid and ggplot2 to allow for rich text formatting in R graphics. [Read more ...](/blog/2024/05/marquee-0-1-0/) 2024/05/29 [![](/blog/2024/03/ggplot2-3-5-0-coord-radial/thumbnail-sq.jpg)](/blog/2024/03/ggplot2-3-5-0-coord-radial/) [ggplot2 3.5.0: Introducing: coord\_radial()](/blog/2024/03/ggplot2-3-5-0-coord-radial/) [package](/categories/package) Teun van den Brand Introducing a new polar coordinate system that supersedes the old `coord_polar()`. Read on about the new `coord_radial()`. [Read more ...](/blog/2024/03/ggplot2-3-5-0-coord-radial/) 2024/03/01 [![](/blog/2024/02/ggplot2-3-5-0-axes/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0-axes/) [ggplot2 3.5.0: Axes](/blog/2024/02/ggplot2-3-5-0-axes/) [package](/categories/package) Teun van den Brand The 3.5.0 version of ggplot2 comes with an overhaul of the guide system. Read here what is new for axes. [Read more ...](/blog/2024/02/ggplot2-3-5-0-axes/) 2024/02/28 [![](/blog/2024/02/ggplot2-3-5-0-legends/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0-legends/) [ggplot2 3.5.0: Legends](/blog/2024/02/ggplot2-3-5-0-legends/) [package](/categories/package) Teun van den Brand The 3.5.0 version of ggplot2 comes with an overhaul of the guide system. Read here what is new for legends. [Read more ...](/blog/2024/02/ggplot2-3-5-0-legends/) 2024/02/26 [![](/blog/2024/02/ggplot2-3-5-0/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0/) [ggplot2 3.5.0](/blog/2024/02/ggplot2-3-5-0/) [package](/categories/package) Teun van den Brand ggplot2 3.5.0 is now on CRAN. Discover what is new in this release. [Read more ...](/blog/2024/02/ggplot2-3-5-0/) 2024/02/23 [![](/blog/2022/11/ggplot2-3-4-0/thumbnail-sq.jpg)](/blog/2022/11/ggplot2-3-4-0/) [ggplot2 3.4.0](/blog/2022/11/ggplot2-3-4-0/) [package](/categories/package) Thomas Lin Pedersen ggplot2 3.4.0 is now on CRAN. Read all about the (mostly internal) changes that make up this release. [Read more ...](/blog/2022/11/ggplot2-3-4-0/) 2022/11/07 [![](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/thumbnail-sq.jpg)](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) [Make your ggplot2 extension package understand the new linewidth aesthetic](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen The next release of ggplot2 will contain a number of internal improvements and fixes long-time inconsistencies. One of these are the conflation of point size and linewidth into the same aesthetic. This post will go into detail with how you can make your extension package work well with the new linewidth aesthetic. [Read more ...](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) 2022/08/24 [![](/blog/2022/04/scales-1-2-0/thumbnail-sq.jpg)](/blog/2022/04/scales-1-2-0/) [scales 1.2.0](/blog/2022/04/scales-1-2-0/) [package](/categories/package) Hadley Wickham scales 1.2.0 brings a number of small but useful improvements to numeric labels. [Read more ...](/blog/2022/04/scales-1-2-0/) 2022/04/13 * [««](/tags/ggplot2/) * « * [1](/tags/ggplot2/) * [2](/tags/ggplot2/page/2/) * [3](/tags/ggplot2/page/3/) * [»](/tags/ggplot2/page/2/) * [»»](/tags/ggplot2/page/3/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Joining the ggplot2 team [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Joining the ggplot2 team ======================== ![](/blog/2025/01/joining-ggplot2/thumbnail-wd.jpg) Photo by [Jonathan Kemper](https://unsplash.com/photos/person-holding-green-plant-on-black-pot-CbZh3kaPxrE)   2025/01/09   [ggplot2](/tags/ggplot2/) , [internship](/tags/internship/)   Teun van den Brand Hello there! I’ve been working on ggplot2 for a while now, and I’d like to tell you how that came about and what it is like. How I got involved[](#how-i-got-involved) ------------------------------------------ My journey into learning R started in 2017 during an internship at the EMBL-EBI. The main gripe about base R plotting that drove me into ggplot2’s arms were the arcane invocations to get anything else than one of the pre-approved chart types. In contrast, ggplot2 absorbs a bunch of small paper cuts, is very compositional in nature while remaining highly customisable. In a bid to “learn from the mistakes of others” rather than (continue to copiously) make my own, I became active on Stack Overflow answering questions and solving plotting issues. For posterity: this was in the days before you could ask an large language model for personalised advice and actual humans were equally frustrated on both sides of the question. I was keeping track of solutions to common problems in a personal cookbook that had its own arcane invocations. To give a bit of flavour: much of the cookbook was about preparing gtables (the data structure that comes out of building a plot) for combining and aligning plots. [1](#fn:1) The cookbook eventually grew into my first ggplot2 extension package: [ggh4x](https://teunbrand.github.io/ggh4x/) . Perhaps that package would be best subtitled: ‘Remedies to my common ggplot2 ailments’. It contains a bunch of miscellaneous functions ranging from reorganising facets to putting minor ticks on the axes. The nature of the package was also its downside, as ggh4x lacked any sense of scope (and still does, as befits any first package). Around the time when I was really getting into ggplot extensions, [Gina Reynolds](https://github.com/EvaMaeRey) had started organising a meeting for people who build ggplot2 extensions. It is an interesting place to meet others and hear about their packages and how they face interacting with the ggplot2 extension system. I started attending with some degree of regularity and made a discussion place on GitHub. We now use this for general exchange of ideas, but also package specific issues. Meanwhile, the questions on Stack Overflow kept directing my attention at the ggplot2 issue tracker every once in a while. After lurking in there for a bit, I started my first informal contributions to ggplot2 itself by answering the simple stuff just as I did on Stack Overflow. It may not seem like much of a contribution, but in retrospect, answering issues helps triaging them: it separates those issues that need additional changes in ggplot2 from those that do not. My first ‘proper contribution’ in the shape of a pull request was in 2020. It replaced 3 lines of code with 2 lines of code to benefit type stability (this was prior to [vctrs](https://vctrs.r-lib.org/) )[2](#fn:2) . In 2022, I commented “I’d be willing to take a stab at this” on an issue proposing a large refactor of the guide system. I like to think it was this precise moment that Thomas, the project lead after having taken over for Hadley, took notice and later invited me to join the team[3](#fn:3) . This new guide system ended up laying the foundation for [legendry](https://teunbrand.github.io/legendry/) , so it wasn’t entirely out of unselfish reasons that I volunteered. At any rate, this is a great opportunity to fill big shoes on a major R project, so I’m very excited to have joined! Becoming an insider[](#becoming-an-insider) -------------------------------------------- Part of being on the team is straightforward. You triage issues. You fix bugs. You implement new features. At the point that I joined, I had already done these things as an outsider. The only thing that really changes is that you get the keys to the kingdom: you can now close issues and merge pull requests [4](#fn:4) . You’re then trusted to wield this power wisely. You then hope you do. At the time I joined the most active maintainers were Thomas, Claus and Hiroaki. I was surprised to learn that really most communication happens on GitHub and it is all public discussion. Even more abstract coordination that does not neatly fit into a single issue, like preparing a new release, didn’t occur behind closed doors. I think what made my introduction to the team more awkward than it needed to be was that GitHub issues is not really a good place for announcements where you can say ‘Hi everyone, this person is on the team now and will be doing stuff in the project’. I had interacted with the other active maintainers before, so I wasn’t a completely alien actor, but I felt some unclarity lingered longer than it ought have. Perhaps I should more assertively have introduced myself [5](#fn:5) . However, by the time posit::conf(2024) was over, I’ve met 6 out of the 9 other authors in person. I have more thoughts about conf and my first time in the United States, but it has been amazing to meet all these people in person whose work you’ve been admiring for a while! Maintaining ggplot2[](#maintaining-ggplot2) -------------------------------------------- The ggplot2 package has both the blessing and the curse of being a popular package. One the one hand, it is a blessing that people care about the project, post issues that they find and make intermittent contributions. The curse is that it is such a staple in the R ecosystem, that almost any change will inadvertently affect somebody else’s code. Not only because ggplot2 is widely used, but also because people have been …creative… with how they are using ggplot2. The art of making changes is to largely affect plots in a good way. The first big project I was rummaging through was the guide system I proposed to rewrite. The guide system had never been advertised as an official extension point, but naturally that didn’t preclude people from using it as an extension point anyway.[6](#fn:6) So in addition to rewriting the system, we also had to prevent terribly breaking extensions that relied on the old system. In some cases, this meant sending out PRs to other packages to be compatible with both systems. Having worked through a good number of issues at this point in time, I can see some emergent patterns. Different patterns can be partially explained by different audiences. The regular user wants to be empowered to execute their vision of a plot effectively. Maintainers of extensions would often like things to work consistently or change a very obscure line somewhere that they have identified as blocking a niche use case. Teachers would like their students to get stuck less often, which often involves improving error messages. All in all, there is no shortage of issues to work through. The next big thing we’re working on is some practical necromancy in getting themeable aesthetics resurrected, which was [initiated by Dana Paige Seidel](https://www.danaseidel.com/2018-09-01-ATidySummer/) all the way back in 2018! We’d like the theme to be a home for more default choices than just non-data elements. Default layer aesthetics are a start, but we plan on putting in default palettes too. A few words of thanks[](#a-few-words-of-thanks) ------------------------------------------------ I’ve been plucked from a level of relative obscurity —a package maintainer that has this weird miscellaneous package— into the path of a flagship R project, for which I’m very grateful. First and foremost I’m thankful to Thomas Lin Pedersen, who has put me into this position and steers the ggplot2 project. Secondly to Hadley Wickham and the rest of the tidyverse team, who make me feel included; both at conf and during regular meetings[7](#fn:7) . Thirdly, the co-authors I met during conf: Claus Wilke, for whose workshop I TA’d, but also Kara Woo and Winston Chang. Lastly, I’d like to thank Posit the company for contracting me to do work I also enjoy as a hobby! * * * 1. Luckily, we don’t have to think about this _at all_, thanks to the [patchwork](https://patchwork.data-imaginist.com/) package! [↩︎](#fnref:1) 2. I’m omitting here that I also had to write 50 lines of tests for this small change [↩︎](#fnref:2) 3. How much this actually reflects any truth is for any of us to guess and for Thomas to know. Later, I learned that this was also [how Thomas himself was roped into the project](https://www.data-imaginist.com/posts/2016-10-31-becoming-the-intern/) ! [↩︎](#fnref:3) 4. After review though. You’re not given _that_ much power! [↩︎](#fnref:4) 5. But I’m not celebrated for my social graces :) [↩︎](#fnref:5) 6. I don’t have a moral high ground here: I was one of the worst offenders! [↩︎](#fnref:6) 7. Mostly for The Golden Hex Sticker though! [↩︎](#fnref:7) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidymodels Internship for 2025 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) tidymodels Internship for 2025 ============================== ![](/blog/2025/01/tidymodels-2025-internship/thumbnail-wd.jpg) Photo by [Jordane Mathieu](https://unsplash.com/photos/black-and-red-pendant-lamp-9rrtd2z5jdg)   2025/01/08   [tidymodels](/tags/tidymodels/) , [internship](/tags/internship/)   Max Kuhn We are chuffed once again to offer a summer internship with the tidymodels team. We’ve had eight previous summer interns and these led to the creation of a number of new packages: [agua](https://agua.tidymodels.org/) , [applicable](https://applicable.tidymodels.org/) , [bundle](https://rstudio.github.io/bundle/) , [butcher](https://butcher.tidymodels.org/) , [shinymodels](https://shinymodels.tidymodels.org/) , [spatialsample](https://spatialsample.tidymodels.org/) , and [stacks](https://stacks.tidymodels.org/) . Our own [Simon Couch](https://www.simonpcouch.com/) is a former intern who won [an award](https://community.amstat.org/jointscsg-section/awards/john-m-chambers) for his work. This year, the primary focus is on expanding our feature selection capabilities. Some of this will involve new recipe steps and other functions. Towards the end of the internship, there might be time to work on other things, too! To apply, make sure that you have a GitHub handle and follow this link: **[`https://posit.co/job-detail/?gh_jid=6323043003`](https://posit.co/job-detail/?gh_jid=6323043003) ** The internship is US-based. If you want to know what the internship is like, a few of our alumni have written about it: * [_A summer with RStudio_ (2018)](https://www.alexpghayes.com/post/2018-08-10_a-summer-with-rstudio/) * [_RStudio Summer Internship_ (2018)](https://fbchow.rbind.io/2018/07/27/rstudio-summer-internship/) * [_This Is Not Like the Others_ (2019)](https://education.rstudio.com/blog/2019/12/this-is-not-like-the-others/) * [_Tidymodels Internship_ (2020)](https://education.rstudio.com/blog/2020/06/tidymodels-internship/) * [_I know what I did last summer_ (2022)](https://www.mm218.dev/posts/2022-08-15-last-summer/) We can’t wait to get started and look forward to reading your applications. The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # wasm [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Wasm [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 [![](/blog/2024/10/webr-0-4-2/thumbnail-sq.jpg)](/blog/2024/10/webr-0-4-2/) [WebAssembly roundup part 1: webR 0.4.2](/blog/2024/10/webr-0-4-2/) [roundup](/categories/roundup) George Stagg First in a series of blog posts about what’s new in R for WebAssembly. The latest release of webR 0.4.2 is now available, with updates to the viewer mechanism, support for displaying htmlwidgets, an IndexedDB virtual filesystem driver, and improved packaging for WebAssembly R binaries. [Read more ...](/blog/2024/10/webr-0-4-2/) 2024/10/11 [![](/blog/2024/04/webr-0-3-1/thumbnail-sq.jpg)](/blog/2024/04/webr-0-3-1/) [webR 0.3.1](/blog/2024/04/webr-0-3-1/) [package](/categories/package) George Stagg webR 0.3.1 is now available at npm, GitHub, and via CDN. Take a look at what’s new in this release. [Read more ...](/blog/2024/04/webr-0-3-1/) 2024/04/02 [![](/blog/2023/08/webr-0-2-0/thumbnail-sq.jpg)](/blog/2023/08/webr-0-2-0/) [webR 0.2.0 has been released](/blog/2023/08/webr-0-2-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.2.0 has been released. Updates and improvements to the webR REPL app, HTML canvas graphics device, internationalisation, Wasm R packages, Shiny support, and the developer API. [Read more ...](/blog/2023/08/webr-0-2-0/) 2023/08/16 [![](/blog/2023/03/webr-0-1-0/thumbnail-sq.jpg)](/blog/2023/03/webr-0-1-0/) [webR 0.1.0 has been released](/blog/2023/03/webr-0-1-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.1.0 has been released! Using the magic of WebAssembly, webR allows you to run R code directly within a web browser. [Read more ...](/blog/2023/03/webr-0-1-0/) 2023/03/09 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # WebAssembly roundup part 2: Shinylive 0.8.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) WebAssembly roundup part 2: Shinylive 0.8.0 =========================================== ![](/blog/2024/10/shinylive-0-8-0/thumbnail-wd.jpg) Photo by [Declan Sun](https://unsplash.com/@declansun)   2024/10/14   [webr](/tags/webr/) , [shiny](/tags/shiny/) , [shinylive](/tags/shinylive/) , [wasm](/tags/wasm/) , [webassembly](/tags/webassembly/)   George Stagg One of the most popular uses of webR in a wider project is [Shinylive](https://shinylive.io/r/examples) , a system for deploying Shiny for R or Python apps that run completely in a web browser, without the need for a dedicated Shiny server. Shinylive works by running both the server and client components in the viewer’s browser, and the support for running R Shiny apps in this way is provided by webR. Since Shinylive works with both R and Python Shiny apps, the project is released as multiple independent but interconnecting software. The core [Shinylive](https://github.com/posit-dev/shinylive) assets, the [R shinylive](https://github.com/posit-dev/r-shinylive) package, the [Python Shinylive](https://github.com/posit-dev/py-shinylive) package, and the [Shinylive Quarto extension](https://github.com/quarto-ext/shinylive/) . This post will describe some the latest changes in the context of running the R Shinylive package and Quarto extension. Shinylive assets[](#shinylive-assets) -------------------------------------- The latest release of the Shinylive assets upgrades the version of webR included to 0.4.2, bringing in the improved packaging and loading performance of R binaries discussed in [part 1 of this series](../webr-0-4-2/) . Shinylive now defaults to downloading R packages in the improved `.tgz` archive format served by the [webR default repository](repo.r-wasm.org) and [R-Universe](https://r-universe.dev/) , resulting in a more efficient R package installation and faster start up process. These changes are already making a tangible difference to applications. In a recent meeting of the [R Consortium Submissions Working Group](https://rconsortium.github.io/submissions-wg/) , it was reported that for a complex Shinylive app the overall load time decreased from over a minute to just 15 seconds! [1](#fn:1) The working group is championing improved practices for R-based submissions of clinical trial data to regulatory bodies for review. With their great work and our steady improvements to Shinylive over time, the group now report that they have reached a new milestone in [successfully submitting a pilot R Shiny app](https://r-consortium.org/posts/using-r-to-submit-research-to-the-fda-pilot-4-successfully-submitted/) , featuring a WebAssembly component with Shinylive, to the FDA for review. [![Screenshots showing the R Consortium Submissions Working Group Pilot 2 Shinylive app.](images/pilot-2.png)](images/pilot-2.png) R shinylive package[](#r-shinylive-package) -------------------------------------------- ### Reproducible data science with binary bundles[](#reproducible-data-science-with-binary-bundles) A benefit of WebAssembly is that the same binary instructions can be executed on a whole range of machine architectures, from high performance desktop workstations to low-power devices such as mobile phones or tablets. WebAssembly provides a common environment ensuring that each device can reproduce the exact same results, both now and potentially for many years into the future. However, those with experience of building software and documents with long-term reproducibility in mind will know that not only must the exact version of your own software be available, but also packages and system dependencies too. Accurate versioning matters; newer editions of R packages are always being released with modified functionality or features deprecated and perhaps even removed. Previously Shinylive downloaded R packages at runtime from the webR default repository. However, that repository follows CRAN and upgrades packages to the latest version reasonably often. So, to help provide long-lived reproducibility, the latest version of Shinylive now not only deploys your application source but also downloads and bundles as many R package binaries as possible in the exported app. By including WebAssembly R package binaries, a self-contained bundle is created that will never change over time, even as new R package versions are released. Once deployed to a static web service such as GitHub Pages or Netlify you can be confident that your results will be exactly the same now or in many years time – at least as long as browsers continue to support the WebAssembly standard! With this, it is now also possible to load a complex R Shinylive app from a local web server without any external internet connection. This isn’t likely to be that useful for most users, but there are some highly regulated industries and restricted network environments where it becomes a key feature. ### Bundling WebAssembly binaries[](#bundling-webassembly-binaries) By default, R packages installed from CRAN, [R-Universe](https://r-universe.dev/) , or [Bioconductor](https://bioconductor.org) will be downloaded and distributed with your Shinylive application. For CRAN packages, the packages are sourced from the webR default repository. For R-Universe or Bioconductor packages, they are sourced from the WebAssembly binaries provided by R-Universe. Here’s an example of what this looks like for a sample Shiny app depending on the dplyr package. Shinylive assets and R package binaries are downloaded and bundled at export time, and the status of each is shown in the output. shinylive::export("app", "site") #> ℹ Exporting Shiny app from: app #> → Destination: site #> [======================================================================] 100% #> ✔ Copying base Shinylive files [289ms] #> ✔ Loading metadata database ... done #> #> Finding R package dependencies ... Done! #> [=======>--------------------------------------------------------------] 11% #> trying URL 'http://repo.r-wasm.org/bin/emscripten/contrib/4.4/dplyr_1.1.4.tgz' #> Content type 'application/x-tar' length 1063948 bytes (1.0 MB) #> ================================================== #> downloaded 1.0 MB #> [...] #> #> ✔ Downloading WebAssembly R package binaries to site/shinylive/webr/packages [3.2s] #> ✔ Writing app metadata to site/shinylive/webr/packages/metadata.rds [14ms] #> ℹ Wrote site/shinylive/webr/packages/metadata.rds (694 bytes) #> ✔ Writing site/app.json [17ms] #> ℹ Wrote site/app.json (1.64K bytes) #> ✔ Shinylive app export complete. #> ℹ Run the following in an R session to serve the app: #> `httpuv::runStaticServer("site")` The shinylive R package will query the currently installed versions of packages on your machine and attempt to download and bundle the same version for WebAssembly. Binaries are considered acceptable if the major and minor version numbers match, and a warning is issued otherwise. This check ensures the resulting behaviour of the exported Shinylive app is as close as possible to the behaviour when running the app in the usual way. shinylive::export("app", "site") #> [...] #> Warning message: #> Package version mismatch for dplyr, ensure the versions below are compatible. #> ! Installed version: 1.0.9, WebAssembly version: 1.1.4. #> ℹ Install a package version matching the WebAssembly version to silence this error. ### Bundling custom R packages[](#bundling-custom-r-packages) Using your own custom R packages with webR or Shinylive is also possible, but requires a little extra work. R packages, particularly those that include compiled code, must be processed specially for WebAssembly. This requires an environment with a WebAssembly compiler toolchain such as Emscripten and some set up to organise the cross-compiling of packages using R. The easiest way to get up and running is to [create a personal R-Universe repository](https://ropensci.org/blog/2021/06/22/setup-runiverse/) for your packages. The system will automatically build R package binaries for multiple targets, including WebAssembly, and Shinylive will download these resulting binaries when exporting your app. It’s also possible to automatically cross-compile and deploy WebAssembly R package binaries using GitHub Actions. The [r-wasm/actions](https://github.com/r-wasm/actions) repository provides reusable workflows for GitHub Actions, one of which can be used to automatically build WebAssembly R package when a GitHub release is created, attaching the resulting binary to the release. If an R package has been installed directly from GitHub, using a tool such as [pak](https://pak.r-lib.org) , Shinylive will look for binaries attached to a GitHub release for bundling. Finally, building an R package for WebAssembly can be done manually using the [rwasm](https://r-wasm.github.io/rwasm/) package. This is a little more involved, using a combination of the [Emscripten SDK](https://github.com/emscripten-core/emsdk) and the [webR Docker container](https://github.com/r-wasm/webr/pkgs/container/webr) to organise cross-compiling packages with R and manage custom CRAN-like repositories. Shinylive will also bundle WebAssembly binaries for R packages installed from such a custom repository. Shinylive Quarto Extension[](#shinylive-quarto-extension) ---------------------------------------------------------- Shinylive applications may be embedded in a Quarto document using the Shinylive Quarto extension. With the extension active, a Shinylive app can be added by directly including its source code in the document markdown. Under the hood, the extension works by calling out to the export functionality provided by the Shinylive R and Python packages, and so improvements to the exporting process also applies to Shiny apps included in Quarto projects. Lorem ipsum dolor sit amet, consectetur adipiscing elit. ```{shinylive-r} #| standalone: true library(shiny) ui <- [...] server <- [...] shinyApp(ui = ui, server = server) ``` Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. ### Embedding data files in subdirectories[](#embedding-data-files-in-subdirectories) When a Shiny app has been deployed with Shinylive it does not have direct access to the filesystem on the client device. This is enforced by WebAssembly and the browser for security reasons. As such, additional data must either by downloaded or pre-loaded to a virtual filesystem before the app starts. There are a few ways to do this with a Shinylive app, but when working in a Quarto document things are more constrained. One supported way is to define the content of additional data files inline. ```{shinylive-r} #| standalone: true ui <- [...] server <- [...] shinyApp(ui = ui, server = server) ## file: data/example.csv foo,bar,baz 1,2,3 2,4,6 3,6,9 5,10,15 8,16,24 ``` The system has been improved to support adding content to subdirectories, along with the ability to define binary content that has been base64 encoded. Combining this with Garrick Aden-Buie’s [quarto-base64](https://github.com/gadenbuie/quarto-base64) extension is a great way to easily include arbitrary data in your Quarto embedded Shinylive apps. ### Quarto project-wide shared assets[](#quarto-project-wide-shared-assets) The latest version of the R shinylive package now checks if the export process is currently running as part of a Quarto render. If so, it uses the `QUARTO_PROJECT_DIR` environment variable as a hint for where to deploy Shinylive assets and bundled WebAssembly R binaries. With this change it’s possible to include multiple Shinylive applications in different documents, sharing their WebAssembly assets across the entire project. This avoids an undesirable situation where the exact same set of fundamental R packages are downloaded and deployed many times to different paths in a Quarto website. Using the latest Shinylive asssets[](#using-the-latest-shinylive-asssets) -------------------------------------------------------------------------- The Shinylive 0.8.0 assets have been [released on GitHub](https://github.com/posit-dev/shinylive/releases/tag/v0.8.0) . They will automatically be downloaded and used once the latest version of the shinylive R package makes it to CRAN and the package has been updated on your machine. If you’d like to get a head start on the latest R shinylive features, you can install the current development version of shinylive directly from GitHub: pak::pak("posit-dev/r-shinylive") Or, if you prefer, you can stick with the current release version of the shinylive R package and orchestrate it to use the latest version of the assets by setting the environment variable: SHINYLIVE_ASSETS_VERSION=0.8.0 Acknowledgements[](#acknowledgements) -------------------------------------- [@chaehni](https://github.com/chaehni) , [@cpsievert](https://github.com/cpsievert) , [@darrida](https://github.com/darrida) , [@erikhall6373](https://github.com/erikhall6373) , [@gadenbuie](https://github.com/gadenbuie) , [@gschivley](https://github.com/gschivley) , [@helgasoft](https://github.com/helgasoft) , [@jeroen](https://github.com/jeroen) , [@JoaoGarcezAurelio](https://github.com/JoaoGarcezAurelio) , [@jvcasillas](https://github.com/jvcasillas) , [@kv9898](https://github.com/kv9898) , [@next-game-solutions](https://github.com/next-game-solutions) , [@Luke-Symes-Tsy](https://github.com/Luke-Symes-Tsy) , [@maek-ies](https://github.com/maek-ies) , [@pawelru](https://github.com/pawelru) , [@quincountychsmn](https://github.com/quincountychsmn) , [@rmcminds](https://github.com/rmcminds) , [@rbcavanaugh](https://github.com/rbcavanaugh) , [@schloerke](https://github.com/schloerke) , [@StefKirsch](https://github.com/StefKirsch) , [@virtualinertia](https://github.com/virtualinertia) , and [@wch](https://github.com/wch) . * * * 1. [https://rconsortium.github.io/submissions-wg/minutes/2024-08-02/#webassembly](https://rconsortium.github.io/submissions-wg/minutes/2024-08-02/#webassembly) [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # webr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Webr [![](/blog/2024/10/quarto-live-0-1-1/thumbnail-sq.jpg)](/blog/2024/10/quarto-live-0-1-1/) [WebAssembly roundup part 3: Quarto Live 0.1.1](/blog/2024/10/quarto-live-0-1-1/) [roundup](/categories/roundup) George Stagg Quarto Live is a new Quarto extension that uses WebAssembly to bring interactive examples and dynamic code exercises to documents. Create highly engaging interactive elements for readers using standard Quarto markdown syntax. [Read more ...](/blog/2024/10/quarto-live-0-1-1/) 2024/10/15 [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 [![](/blog/2024/10/webr-0-4-2/thumbnail-sq.jpg)](/blog/2024/10/webr-0-4-2/) [WebAssembly roundup part 1: webR 0.4.2](/blog/2024/10/webr-0-4-2/) [roundup](/categories/roundup) George Stagg First in a series of blog posts about what’s new in R for WebAssembly. The latest release of webR 0.4.2 is now available, with updates to the viewer mechanism, support for displaying htmlwidgets, an IndexedDB virtual filesystem driver, and improved packaging for WebAssembly R binaries. [Read more ...](/blog/2024/10/webr-0-4-2/) 2024/10/11 [![](/blog/2024/04/webr-0-3-1/thumbnail-sq.jpg)](/blog/2024/04/webr-0-3-1/) [webR 0.3.1](/blog/2024/04/webr-0-3-1/) [package](/categories/package) George Stagg webR 0.3.1 is now available at npm, GitHub, and via CDN. Take a look at what’s new in this release. [Read more ...](/blog/2024/04/webr-0-3-1/) 2024/04/02 [![](/blog/2023/08/webr-0-2-0/thumbnail-sq.jpg)](/blog/2023/08/webr-0-2-0/) [webR 0.2.0 has been released](/blog/2023/08/webr-0-2-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.2.0 has been released. Updates and improvements to the webR REPL app, HTML canvas graphics device, internationalisation, Wasm R packages, Shiny support, and the developer API. [Read more ...](/blog/2023/08/webr-0-2-0/) 2023/08/16 [![](/blog/2023/03/webr-0-1-0/thumbnail-sq.jpg)](/blog/2023/03/webr-0-1-0/) [webR 0.1.0 has been released](/blog/2023/03/webr-0-1-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.1.0 has been released! Using the magic of WebAssembly, webR allows you to run R code directly within a web browser. [Read more ...](/blog/2023/03/webr-0-1-0/) 2023/03/09 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # shiny [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Shiny [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # s7 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) S7 [![](/blog/2024/11/s7-0-2-0/thumbnail-sq.jpg)](/blog/2024/11/s7-0-2-0/) [S7 0.2.0](/blog/2024/11/s7-0-2-0/) [package](/categories/package) Tomasz Kalinowski and Hadley Wickham S7 is a new package that simplifies object-oriented programming (OOP) in R. It combines the simplicity of S3 with the structure of S4 to create a clearer system that’s accessible to everyone. [Read more ...](/blog/2024/11/s7-0-2-0/) 2024/11/07 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # S7 0.2.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) S7 0.2.0 ======== ![](/blog/2024/11/s7-0-2-0/thumbnail-wd.jpg) Photo by [Tomasz Kalinowski](https://github.com/t-kalinowski)   2024/11/07   [s7](/tags/s7/)   Tomasz Kalinowski and Hadley Wickham We’re excited to announce that [S7](https://rconsortium.github.io/S7/) v0.2.0 is now available on CRAN! S7 is a new object-oriented programming (OOP) system designed to supersede both S3 and S4. You might wonder why R needs a new OOP system when we already have two. The reason lies in the history of R’s OOP journey: S3 is a simple and effective system for single dispatch, while S4 adds formal class definitions and multiple dispatch, but at the cost of complexity. This has forced developers to choose between the simplicity of S3 and the sophistication of S4. The goal of S7 is to unify the OOP landscape by building on S3’s existing dispatch system and incorporating the most useful features of S4 (along with some new ones), all with a simpler syntax. S7’s design and implementation have been a collaborative effort by a working group from the [R Consortium](https://www.r-consortium.org) , including representatives from R-Core, Bioconductor, tidyverse/Posit, ROpenSci, and the wider R community. Since S7 builds on S3, it is fully compatible with existing S3-based code. It’s also been thoughtfully designed to work with S4, and as we learn more about the challenges of transitioning from S4 to S7, we’ll continue to add features to ease this process. Our long-term goal is to include S7 in base R, but for now, you can install it from CRAN: install.packages("S7") What’s new in the second release[](#whats-new-in-the-second-release) --------------------------------------------------------------------- The second release of S7 brings refinements and bug fixes. Highlights include: * Support for lazy property defaults, making class setup more flexible. * Custom property setters now run on object initialization. * Significant speed improvements for setting and getting properties with `@` and `@<-`. * Expanded compatibility with base S3 classes. * [`convert()`](https://rconsortium.github.io/S7/reference/convert.html) now provides a default method for transforming a parent class into a subclass. Additionally, there are numerous bug fixes and quality-of-life improvements, such as better error messages, improved support for base Ops methods, and compatibility improvements for using `@` in R versions prior to 4.3. You can see a full list of changes in the [release notes](https://github.com/RConsortium/S7/blob/main/NEWS.md) . Who should use S7[](#who-should-use-s7) ---------------------------------------- S7 is a great fit for R users who like to try new things but don’t need to be the first. It’s already used in several CRAN packages, and the tidyverse team is applying it in new projects. While you may still run into a few issues, many early problems have been resolved. Usage[](#usage) ---------------- library(S7) Let’s dive into the basics of S7. To learn more, check out the package vignettes, including a more detailed introduction in [`vignette("S7")`](https://rconsortium.github.io/S7/articles/S7.html) , and coverage of generics and methods in [`vignette("generics-methods")`](https://rconsortium.github.io/S7/articles/generics-methods.html) , and classes and objects in [`vignette("classes-objects")`](https://rconsortium.github.io/S7/articles/classes-objects.html) . ### Classes and objects[](#classes-and-objects) S7 classes have formal definitions, specified by [`new_class()`](https://rconsortium.github.io/S7/reference/new_class.html) , which includes a list of properties and an optional validator. For example, the following code creates a `Range` class with `start` and `end` properties, and a validator to ensure that `start` is always less than `end`: Range <- new_class("Range", properties = list( start = class_double, end = class_double ), validator = function(self) { if (length(self@start) != 1) { "@start must be length 1" } else if (length(self@end) != 1) { "@end must be length 1" } else if (self@end < self@start) { "@end must be greater than or equal to @start" } } ) [`new_class()`](https://rconsortium.github.io/S7/reference/new_class.html) returns the class object, which also serves as the constructor to create instances of the class: x <- Range(start = 1, end = 10) x #> #> @ start: num 1 #> @ end : num 10 ### Properties[](#properties) The data an object holds are called its **properties**. Use `@` to get and set properties: x@start #> [1] 1 x@end <- 20 x #> #> @ start: num 1 #> @ end : num 20 Properties are automatically validated against the type declared in [`new_class()`](https://rconsortium.github.io/S7/reference/new_class.html) (in this case, `double`) and checked by the class **validator**: x@end <- "x" #> Error: @end must be , not x@end <- -1 #> Error: object is invalid: #> - @end must be greater than or equal to @start ### Generics and methods[](#generics-and-methods) Like S3 and S4, S7 uses **functional OOP**, where methods belong to **generic** functions, and method calls look like regular function calls: `generic(object, arg2, arg3)`. A generic uses the types of its arguments to automatically pick the appropriate method implementation. You can create a new generic with [`new_generic()`](https://rconsortium.github.io/S7/reference/new_generic.html) , specifying the arguments to dispatch on: inside <- new_generic("inside", "x") To define a method for a specific class, use `method(generic, class) <- implementation`: method(inside, Range) <- function(x, y) { y >= x@start & y <= x@end } inside(x, c(0, 5, 10, 15)) #> [1] FALSE TRUE TRUE TRUE Printing the generic shows its methods: inside #> inside(x, ...) with 1 methods: #> 1: method(inside, Range) And you can retrieve the method for a specific class: method(inside, Range) #> method(inside, Range) #> function (x, y) #> { #> y >= x@start & y <= x@end #> } Known limitations[](#known-limitations) ---------------------------------------- While we are pleased with S7’s design, there are still some limitations: * S7 objects can be serialized to disk (with [`saveRDS()`](https://rdrr.io/r/base/readRDS.html) ), but the current implementation saves the entire class specification with each object. This may change in the future. * Support for implicit S3 classes `"array"` and `"matrix"` is still in development. We expect the community will uncover more issues as S7 is more widely adopted. If you encounter any problems, please file an issue at [https://github.com/RConsortium/OOP-WG/issues](https://github.com/RConsortium/OOP-WG/issues) . We appreciate your feedback in helping us make S7 even better! 😃 Acknowledgements[](#acknowledgements) -------------------------------------- Thank you to all people who have contributed issues, code, and comments to this release: [@calderonsamuel](https://github.com/calderonsamuel) , [@Crosita](https://github.com/Crosita) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dipterix](https://github.com/dipterix) , [@guslipkin](https://github.com/guslipkin) , [@gvelasq](https://github.com/gvelasq) , [@hadley](https://github.com/hadley) , [@jeffkimbrel](https://github.com/jeffkimbrel) , [@jl5000](https://github.com/jl5000) , [@jmbarbone](https://github.com/jmbarbone) , [@jmiahjones](https://github.com/jmiahjones) , [@jonthegeek](https://github.com/jonthegeek) , [@JosiahParry](https://github.com/JosiahParry) , [@jtlandis](https://github.com/jtlandis) , [@lawremi](https://github.com/lawremi) , [@MarcellGranat](https://github.com/MarcellGranat) , [@mikmart](https://github.com/mikmart) , [@mmaechler](https://github.com/mmaechler) , [@mynanshan](https://github.com/mynanshan) , [@rikivillalba](https://github.com/rikivillalba) , [@sjcowtan](https://github.com/sjcowtan) , [@t-kalinowski](https://github.com/t-kalinowski) , [@teunbrand](https://github.com/teunbrand) , and [@waynelapierre](https://github.com/waynelapierre) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # other [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Other [![](/blog/2025/01/joining-ggplot2/thumbnail-sq.jpg)](/blog/2025/01/joining-ggplot2/) [Joining the ggplot2 team](/blog/2025/01/joining-ggplot2/) [other](/categories/other) Teun van den Brand I joined the ggplot2 team and would like to share the experience. [Read more ...](/blog/2025/01/joining-ggplot2/) 2025/01/09 [![](/blog/2025/01/tidymodels-2025-internship/thumbnail-sq.jpg)](/blog/2025/01/tidymodels-2025-internship/) [tidymodels Internship for 2025](/blog/2025/01/tidymodels-2025-internship/) [other](/categories/other) Max Kuhn The tidymodels team is sponsoring a summer internship in 2025. [Read more ...](/blog/2025/01/tidymodels-2025-internship/) 2025/01/08 [![](/blog/2024/02/tidymodels-2024-survey/thumbnail-sq.jpg)](/blog/2024/02/tidymodels-2024-survey/) [Take the tidymodels survey for 2024 priorities](/blog/2024/02/tidymodels-2024-survey/) [other](/categories/other) Emil Hvitfeldt We are conducting our third tidymodels priorities survey. Please give us your feedback! [Read more ...](/blog/2024/02/tidymodels-2024-survey/) 2024/02/28 [![](/blog/2023/06/spring-cleaning-2023/thumbnail-sq.jpg)](/blog/2023/06/spring-cleaning-2023/) [Package spring cleaning](/blog/2023/06/spring-cleaning-2023/) [other](/categories/other) Andy Teucher When Spring comes around, it’s time to embark on some Spring Cleaning to take care of the package maintenance tasks that you never seem to get around to. This post outlines the processes and tools that the tidyverse team uses to make this job fun and efficient. [Read more ...](/blog/2023/06/spring-cleaning-2023/) 2023/06/07 [![](/blog/2023/06/code-review-principles/thumbnail-sq.jpg)](/blog/2023/06/code-review-principles/) [Tidyteam code review principles](/blog/2023/06/code-review-principles/) [other](/categories/other) Davis Vaughan We’ve written a collection of tidyteam code review principles that act as a resource for new contributors and as a source of truth when there are questions about our code review process. We hope you find them useful! [Read more ...](/blog/2023/06/code-review-principles/) 2023/06/05 [![](/blog/2023/04/base-vs-magrittr-pipe/thumbnail-sq.jpg)](/blog/2023/04/base-vs-magrittr-pipe/) [Differences between the base R and magrittr pipes](/blog/2023/04/base-vs-magrittr-pipe/) [other](/categories/other) Hadley Wickham A discussion of the (relatively minor) differences between the native R pipe, `|>`, and the magrittr pipe, `%>%`. [Read more ...](/blog/2023/04/base-vs-magrittr-pipe/) 2023/04/21 [![](/blog/2022/09/devrel-hiring-2022/thumbnail-sq.jpg)](/blog/2022/09/devrel-hiring-2022/) [Come work with us in Developer Relations](/blog/2022/09/devrel-hiring-2022/) [other](/categories/other) Tracy Teal We’re hiring a Developer Educator for tidyverse package development and a Quarto Developer Advocate. Come work with us! [Read more ...](/blog/2022/09/devrel-hiring-2022/) 2022/09/22 [![](/blog/2021/12/relicensing-packages/thumbnail-sq.jpg)](/blog/2021/12/relicensing-packages/) [Re-licensing packages: a retrospective](/blog/2021/12/relicensing-packages/) [other](/categories/other) Mara Averick Over the past year and change we re-licensed the vast majority of tidyverse, tidymodels, and r-lib packages to use the MIT license. Here, we discuss the mechanics and rationale. [Read more ...](/blog/2021/12/relicensing-packages/) 2021/12/07 [![](/blog/2021/10/tidymodels-2022-survey/thumbnail-sq.jpg)](/blog/2021/10/tidymodels-2022-survey/) [Take the tidymodels survey for 2022 priorities](/blog/2021/10/tidymodels-2022-survey/) [other](/categories/other) Max Kuhn We are conducting our second tidymodels priorities survey. Please give us your feedback! [Read more ...](/blog/2021/10/tidymodels-2022-survey/) 2021/10/06 [![](/blog/2020/02/tidy-dev-days-2020/thumbnail-sq.jpg)](/blog/2020/02/tidy-dev-days-2020/) [More 2020 tidy dev days!](/blog/2020/02/tidy-dev-days-2020/) [other](/categories/other) Mara Averick A roundup of our post-rstudio::conf tidy dev day, and announcement of a pre-useR! 2020 event. [Read more ...](/blog/2020/02/tidy-dev-days-2020/) 2020/02/28 * [««](/categories/other/) * « * [1](/categories/other/) * [2](/categories/other/page/2/) * [3](/categories/other/page/3/) * [»](/categories/other/page/2/) * [»»](/categories/other/page/3/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # postprocessing [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Postprocessing [![](/blog/2024/10/postprocessing-preview/thumbnail-sq.jpg)](/blog/2024/10/postprocessing-preview/) [Postprocessing is coming to tidymodels](/blog/2024/10/postprocessing-preview/) [roundup](/categories/roundup) Simon Couch, Hannah Frick, and Max Kuhn The tidymodels team has been hard at work on postprocessing, a set of features to adjust model predictions. The functionality includes a new package as well as changes across the framework. [Read more ...](/blog/2024/10/postprocessing-preview/) 2024/10/08 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # quarto [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Quarto [![](/blog/2024/10/quarto-live-0-1-1/thumbnail-sq.jpg)](/blog/2024/10/quarto-live-0-1-1/) [WebAssembly roundup part 3: Quarto Live 0.1.1](/blog/2024/10/quarto-live-0-1-1/) [roundup](/categories/roundup) George Stagg Quarto Live is a new Quarto extension that uses WebAssembly to bring interactive examples and dynamic code exercises to documents. Create highly engaging interactive elements for readers using standard Quarto markdown syntax. [Read more ...](/blog/2024/10/quarto-live-0-1-1/) 2024/10/15 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # workflows [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Workflows [![](/blog/2024/10/postprocessing-preview/thumbnail-sq.jpg)](/blog/2024/10/postprocessing-preview/) [Postprocessing is coming to tidymodels](/blog/2024/10/postprocessing-preview/) [roundup](/categories/roundup) Simon Couch, Hannah Frick, and Max Kuhn The tidymodels team has been hard at work on postprocessing, a set of features to adjust model predictions. The functionality includes a new package as well as changes across the framework. [Read more ...](/blog/2024/10/postprocessing-preview/) 2024/10/08 [![](/blog/2024/04/tidymodels-2024-q1/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-2024-q1/) [Q1 2024 tidymodels digest](/blog/2024/04/tidymodels-2024-q1/) [roundup](/categories/roundup) Hannah Frick The tidymodels team has been busy working on all sorts of new features across the framework. [Read more ...](/blog/2024/04/tidymodels-2024-q1/) 2024/04/24 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2021/07/tidymodels-july-2021/thumbnail-sq.jpg)](/blog/2021/07/tidymodels-july-2021/) [New tidymodels releases for July 2021](/blog/2021/07/tidymodels-july-2021/) [roundup](/categories/roundup) Max Kuhn We highlight a series of new tidymodels package versions and their improvements. [Read more ...](/blog/2021/07/tidymodels-july-2021/) 2021/07/27 [![](/blog/2021/05/choose-tidymodels-adventure/thumbnail-sq.jpg)](/blog/2021/05/choose-tidymodels-adventure/) [Choose your own tidymodels adventure](/blog/2021/05/choose-tidymodels-adventure/) [learn](/categories/learn) Julia Silge The tidymodels ecosystem is modular and flexible, which can sometimes make choosing an approach overwhelming for newcomers. This post offers opinionated guidance on where to start! [Read more ...](/blog/2021/05/choose-tidymodels-adventure/) 2021/05/24 [![](/blog/2021/03/workflowsets-0-0-1/thumbnail-sq.jpg)](/blog/2021/03/workflowsets-0-0-1/) [workflowsets 0.0.1](/blog/2021/03/workflowsets-0-0-1/) [package](/categories/package) Max Kuhn A new package for evaluating many models at once is now on CRAN, along with a corresponding update to the tidyposterior package. [Read more ...](/blog/2021/03/workflowsets-0-0-1/) 2021/03/30 [![](/blog/2020/11/tune-0-1-2/thumbnail-sq.jpg)](/blog/2020/11/tune-0-1-2/) [tune 0.1.2](/blog/2020/11/tune-0-1-2/) [package](/categories/package) Max Kuhn A new version of the tune package contains numerous new features. [Read more ...](/blog/2020/11/tune-0-1-2/) 2020/11/23 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # shinylive [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Shinylive [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # WebAssembly roundup part 3: Quarto Live 0.1.1 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) WebAssembly roundup part 3: Quarto Live 0.1.1 ============================================= ![](/blog/2024/10/quarto-live-0-1-1/thumbnail-wd.jpg) Photo by [Alexander Grey](https://unsplash.com/@sharonmccutcheon)   2024/10/15   [quarto](/tags/quarto/) , [webr](/tags/webr/)   George Stagg We’re tickled pink to announce the release of [Quarto Live](https://r-wasm.github.io/quarto-live/) 0.1.1. Quarto Live is a new Quarto extension that uses WebAssembly to bring interactive examples and code exercises with custom grading algorithms to your HTML-based output documents, using standard Quarto markdown syntax. Quarto Live adds a [CodeMirror](https://codemirror.net/) \-based text editor to your document with automatic theming, syntax highlighting, and auto-complete. The editor executes R code using webR, and even integrates with [Quarto’s OJS support](https://quarto.org/docs/interactive/ojs/) so that interactive code cells update reactively with other `ojs` cells running in the page. This blog post is part 3 of a WebAssembly roundup series, and will discuss Quarto Live’s primary features and show some examples of the extension in use. Authors who are creating educational content should find this post particularly interesting, as adding just a little interactivity can go a long way to keep readers engaged. The post contains only static screenshots, but if you’d like see Quarto Live in action there are interactive examples throughout its [documentation website](https://r-wasm.github.io/quarto-live/) . Getting Started[](#getting-started) ------------------------------------ You can add the Quarto Live extension to a project by running the following command in a terminal with the current directory of a Quarto project: quarto add r-wasm/quarto-live Then, create a new document with the following template to get up and running using the `knitr` engine: --- title: R Example engine: knitr format: live-html --- {{< include ./_extensions/r-wasm/live/_knitr.qmd >}} ## Main Content Once the document has been set up in this way, an R code block can be made into an interactive code block simply by switching `{r}` to `{webr}`. In this example, we create an interactive block that plots example data using the ggplot package. ```{webr} #| warning: false library(ggplot2) ggplot(airquality, aes(Temp, Ozone)) + geom_point() + geom_smooth(method = "loess") ``` The resulting rendered document looks like the screenshot below. An editor is inserted into the document in the place of the code block, and an interested reader can use it to modify and re-execute the source code in place. ![Screenshot showing the result of rendering the example document above. The Quarto Live editor is shown pre-populated with the provided code snippet. An output graphic is shown, showing the result of plotting the airquality dataset using the ggplot2 package.](images/ggplot-1.png) Quarto Live executes R code using the [evaluate](https://evaluate.r-lib.org) package, with output rendered using functions from [knitr](https://yihui.org/knitr/) , so the output should be almost identical to output generated by R Markdown or Quarto. The real beauty of this, in my opinion, is that the reader does not have to install any packages, copy and paste source code, switch to an IDE like [RStudio](https://posit.co/products/open-source/rstudio/) or [Positron](https://positron.posit.co) , or deal with myriad other small but fiddly distractions just to experiment with a new piece of code or R package. They can do it, right there, without any context switching. At first you might think of this like cells in a Jupyter notebook. However, to me a notebook feels more like an exploratory environment, whereas a Quarto Live block feels more like published content; it lives somewhere in-between computational notebooks and the static rendered output of literate programming frameworks like R Markdown and Quarto. Interactive exercises[](#interactive-exercises) ------------------------------------------------ Traditionally, this level of direct interactivity for R code in a rendered document has only been possible though tools that require a server-side component, such as a Jupyter server or a Shiny server using the [learnr](https://rstudio.github.io/learnr/) package to execute code dynamically. This has limited deployment options for educators, particularly those who are restricted in where they can deploy to an institution’s own [learning management system (LMS)](https://en.wikipedia.org/wiki/Learning_management_system) , or hindered by the sheer number of clients in the case of extremely large class sizes. The rise of virtual learning over the last few years has exacerbated the problem, tutorials might no longer be in-person in a managed computer lab, but virtually over the internet and on entirely student controlled devices. WebAssembly brings a potential solution to this problem in the form of a universal runtime with minimal dependencies. Using Quarto Live an interactive tutorial can be rendered into static HTML output that’s well supported by third party virtual learning environments (when compared to traditional Shiny apps) without ongoing management of a server component. ### Defining an exercise[](#defining-an-exercise) Here’s an example showing how to create an interactive tutorial using Quarto Live. We’ll build an exercise with a grading component, so that visitors can get feedback on their responses. ### Exercise 1 Calculate the average of all the integers in the vector defined as the variable `foo`. ```{webr} #| exercise: ex_1 ______(foo) ``` This will add an interactive code editor to the page, along with some placeholder code. Notice how the placeholder contains a string of six underscore (`_`) characters. When defining an exercise, Quarto Live will consider six or more underscores as a “blank” that must be replaced by the learner. ![Screenshot showing the result of rendering the example exercise above. The Quarto Live editor is shown pre-populated with the placeholder code. An error message is shown: Please replace ______ with valid code.](images/blank.png) In the exercise we’ve asked about a variable `foo`, but not created it anywhere yet. Let’s fix that by adding a `setup` block that will always be executed before learner submitted code. This block can appear before or after the one above, the placement does not matter as it’s linked to the exercise by it’s label. ```{webr} #| exercise: ex_1 #| setup: true foo <- sample.int(100, 10) ``` ![Animation showing the result of rendering the example above. The Quarto Live editor is shown pre-populated with the code `foo`. A button labelled 'Run Code' is pressed repeatedly and the output changes each time. The output consists of 10 random integers.](images/sample.gif) A `check` code block defines a grading algorithm, checking submitted code and assigning feedback in the form of a [feedback list](https://r-wasm.github.io/quarto-live/exercises/grading.html#return-feedback) . ```{webr} #| exercise: ex_1 #| check: true if (identical(.result, mean(foo))) { list(correct = TRUE, message = "Nice work!") } else { list(correct = FALSE, message = "That's incorrect, sorry.") } ``` ![Screenshots showing the result of rendering the example exercise above. The editor is first shown with correct code. A success message is shown: Nice work!. A second editor shows the incorrect code. A failure message is shown: That's incorrect, sorry. ](images/exercise.png) Finally, let’s add some solution text. This time we’ll use a Quarto fenced block to define the content. We still link this block to our exercise by providing the label we used before, just with a slightly different syntax. When a “hint” or “solution” block is added in this way, it is hidden until requested to be revealed by the learner through the Quarto Live exercise editor UI. ::: { .solution exercise="ex_1" } ::: { .callout-tip title="Solution" collapse="false"} Here is a possible solution: ```r bar <- mean(foo) #<1> print(bar) #<2> ``` 1. Use the `mean` function with `foo` to calculate the average, store this value as `bar`. 2. Print the value stored in `bar` to the console. ::: ::: ![Screenshot showing the result of adding the solution block above. The solution has been revealed, demonstrating the callout block and code annotation features.](images/solution.png) One really great thing about the way this works is that content is defined using standard Quarto markdown syntax. That means we can take full advantage of all the great features that Quarto provides for describing source code and results. Features like collapsible callout blocks and annotated source code allow us to present hints and solutions in the most effective way for learners. You can read more about exercises and grading, including examples using the existing [gradethis](https://pkgs.rstudio.com/gradethis/index.html) package, in the [Quarto Live documentation](https://r-wasm.github.io/quarto-live/exercises/grading.html) . Reactivity with OJS[](#reactivity-with-ojs) -------------------------------------------- Quarto Live cells may define or take input from OJS reactive variables in the page, providing a seamless way to create dynamic experiences without requiring the use of R Shiny. It is this technology that powers the grading feature shown in the previous section. In the following example a Quarto Live cell takes input from an OJS variable and defines an output OJS variable. Notice how updates in the Quarto Live cell are propagated to related `ojs` cells in the page. ```{ojs} foo = 123; ``` ```{ojs} bar ``` ```{webr} #| input: ['foo'] #| define: ['bar'] bar <- foo ** 2 ``` ![Animation showing the result of rendering the example above. The Quarto Live editor and several OJS cells are shown. The code is modified and 'Run Code' is pressed several times. The output OJS cell reactively updates with each execution.](images/ojs.gif) You can even define a function in R and then invoke it reactively using an `ojs` cell. JavaScript arguments will be converted into R objects, including transparently handling datasets using webR’s generic R object constructor described in the [first post](/blog/2024/10/webr-0-4-2/) of this blog series. In this example an R function is defined that produces some output using base plotting commands. The function is executed from an `ojs` cell, reactively in response to a changing OJS input. ```{webr} #| include: false #| define: draw_hist draw_hist <- function(colour) { hist(rnorm(1000), col = colour) } ``` ```{ojs} //| echo: false viewof colour = Inputs.select( [ 'orangered', 'forestgreen', 'cornflowerblue' ], { label: 'Colour' } ); draw_hist(colour); ``` ![Animation showing the result of rendering the example above. A histogram is shown below a dropdown options menu offering a selection of colours. As colours are selected, the histogram is redrawn using that colour as a fill.](images/hist.gif) We hope this form of reactivity will become a powerful pattern to create rich interactive experiences for readers. For more examples of integration with OJS, take a look at the [penguins dashboard-like plot example](https://r-wasm.github.io/quarto-live/interactive/reactivity.html#overview) and [dynamic exercises](https://r-wasm.github.io/quarto-live/interactive/dynamic.html) in the Quarto Live documentation. Displaying htmlwidgets[](#displaying-htmlwidgets) -------------------------------------------------- The popular [htmltools](https://rstudio.github.io/htmltools/) and [htmlwidgets](https://www.htmlwidgets.org/) packages bring HTML and JavaScript widgets to R, and thanks to updates in webR such HTML output can also be displayed by Quarto Live. Simply print a HTML object or a widget in a live code block and the result will be dynamically added to the web page. ![Animation showing the result of rendering the example above. A histogram is shown below a dropdown options menu offering a selection of colours. As colours are selected, the histogram is redrawn using that colour as a fill.](images/leaflet.png) One more thing[](#one-more-thing) ---------------------------------- By the way, everything I’ve shown in this blog post also works for Python using the [Pyodide](https://pyodide.org) WebAssembly engine. Pyodide works really well for executing Python code on the web and inspired much of how the webR engine and library works today. Many examples of using Quarto Live to evaluate Python code, including dynamic experiences similar to those shown the previous sections, can be found on the [Quarto Live documentation website](https://r-wasm.github.io/quarto-live/) . Acknowledgements[](#acknowledgements) -------------------------------------- I’m excited and fascinated to see Quarto Live start being used by the community to create interactive content for the education space and beyond. The project is still fairly new, but we would already not where we are without the help of early users providing their comments, issues and bug reports. Thank you! [@Analect](https://github.com/Analect) , [@andrewpbray](https://github.com/andrewpbray) , [@aneesha](https://github.com/aneesha) , [@arnaud-feldmann](https://github.com/arnaud-feldmann) , [@coatless](https://github.com/coatless) , [@cwickham](https://github.com/cwickham) , [@CyuHat](https://github.com/CyuHat) , [@DrDeception](https://github.com/DrDeception) , [@fcichos](https://github.com/fcichos) , [@joelnitta](https://github.com/joelnitta) , [@joelostblom](https://github.com/joelostblom) , [@kcarnold](https://github.com/kcarnold) , [@michaelplynch](https://github.com/michaelplynch) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@Nenuial](https://github.com/Nenuial) , [@rpruim](https://github.com/rpruim) , [@rundel](https://github.com/rundel) , [@ryjohnson09](https://github.com/ryjohnson09) , and [@tmieno2](https://github.com/tmieno2) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # webassembly [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Webassembly [![](/blog/2024/10/shinylive-0-8-0/thumbnail-sq.jpg)](/blog/2024/10/shinylive-0-8-0/) [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) [roundup](/categories/roundup) George Stagg Shinylive deploys Shiny apps that run completely in the browser. Shinylive 0.8.0 has now been released, with a bunch of bug-fixes, changes to better support multiple apps in a Quarto project, and a new system for including package binaries as part of a fully self-contained bundle. [Read more ...](/blog/2024/10/shinylive-0-8-0/) 2024/10/14 [![](/blog/2024/04/webr-0-3-1/thumbnail-sq.jpg)](/blog/2024/04/webr-0-3-1/) [webR 0.3.1](/blog/2024/04/webr-0-3-1/) [package](/categories/package) George Stagg webR 0.3.1 is now available at npm, GitHub, and via CDN. Take a look at what’s new in this release. [Read more ...](/blog/2024/04/webr-0-3-1/) 2024/04/02 [![](/blog/2023/08/webr-0-2-0/thumbnail-sq.jpg)](/blog/2023/08/webr-0-2-0/) [webR 0.2.0 has been released](/blog/2023/08/webr-0-2-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.2.0 has been released. Updates and improvements to the webR REPL app, HTML canvas graphics device, internationalisation, Wasm R packages, Shiny support, and the developer API. [Read more ...](/blog/2023/08/webr-0-2-0/) 2023/08/16 [![](/blog/2023/03/webr-0-1-0/thumbnail-sq.jpg)](/blog/2023/03/webr-0-1-0/) [webR 0.1.0 has been released](/blog/2023/03/webr-0-1-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.1.0 has been released! Using the magic of WebAssembly, webR allows you to run R code directly within a web browser. [Read more ...](/blog/2023/03/webr-0-1-0/) 2023/03/09 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # patchwork [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Patchwork [![](/blog/2024/09/patchwork-1-3-0/thumbnail-sq.jpg)](/blog/2024/09/patchwork-1-3-0/) [patchwork 1.3.0](/blog/2024/09/patchwork-1-3-0/) [package](/categories/package) Thomas Lin Pedersen patchwork 1.3.0 has just been released bringing refinements to the `free()` function and full on support for gt tables [Read more ...](/blog/2024/09/patchwork-1-3-0/) 2024/09/13 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # gt [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Gt [![](/blog/2024/09/patchwork-1-3-0/thumbnail-sq.jpg)](/blog/2024/09/patchwork-1-3-0/) [patchwork 1.3.0](/blog/2024/09/patchwork-1-3-0/) [package](/categories/package) Thomas Lin Pedersen patchwork 1.3.0 has just been released bringing refinements to the `free()` function and full on support for gt tables [Read more ...](/blog/2024/09/patchwork-1-3-0/) 2024/09/13 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # patchwork 1.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) patchwork 1.3.0 =============== ![](/blog/2024/09/patchwork-1-3-0/thumbnail-wd.jpg) Photo by [Dihn Pham](https://unsplash.com/photos/sewing-silk-and-scissors-dG35-kUxv34)   2024/09/13   [ggplot2](/tags/ggplot2/) , [gt](/tags/gt/) , [patchwork](/tags/patchwork/)   Thomas Lin Pedersen I’m excited to present [patchwork](https://patchwork.data-imaginist.com) 1.3.0, our package for creating multifigure plot compositions. This versions adds table support and improves support for “free"ing components to span across multiple grid cells. You can install patchwork from CRAN with: install.packages("patchwork") You can see a full list of changes in the [release notes](https://patchwork.data-imaginist.com/news/index.html) library(patchwork) library(ggplot2) library(gt) Tables are figures too[](#tables-are-figures-too) -------------------------------------------------- The new and shiny feature of the release is that patchwork now has native support for gt objects, making it possible to compose beautifully formatted tables together with your figures. This has been made possible through Teun Van den Brand’s effort to provide grob output to gt. While this means that you can now pass in gt objects to [`wrap_elements()`](https://patchwork.data-imaginist.com/reference/wrap_elements.html) in the same way as other supported data types, it also goes one step further, using the semantics of the table design to add table specific formatting options through the new [`wrap_table()`](https://patchwork.data-imaginist.com/reference/wrap_table.html) function. But let’s take a step back and see how the simplest support works in reality: p1 <- ggplot(airquality) + geom_line(aes(x = Day, y = Temp, colour = month.name[Month])) + labs(colour = "Month") aq <- airquality[sample(nrow(airquality), 10), ] p1 + gt(aq) + ggtitle("Sample of the dataset") ![](figs/unnamed-chunk-2-1.png) A few things can be gathered already from this small example. Tables can have titles (and subtitles, captions, and tags) like regular plots (in that sense they behave like [`wrap_elements()`](https://patchwork.data-imaginist.com/reference/wrap_elements.html) output). Also, and this is perhaps more interesting, patchwork is aware that the first row is special (a header row), and thus places that on top of the panel area so that the plot region of the left plot is aligned with the body of the table, not the full table. Lastly, we see that tables often have a fixed size, contrary to plots which can shrink and expand based on how much room they have. Because of this, our table is overflowing it’s region in the plot above creating a not-so-great look. Let’s see how we can use [`wrap_table()`](https://patchwork.data-imaginist.com/reference/wrap_table.html) to control some of these behaviors. First, while we could decrease the font size in the table to make it smaller, we could also allow it some more space instead. We could do this by using `plot_layout(widths = ...)` but it would require a fair amount of guessing on our side to get it just right. Thankfully, patchwork is smart enough to figure it out for us and we can instruct it to do so using the `space` argument in [`wrap_table()`](https://patchwork.data-imaginist.com/reference/wrap_table.html) . Setting it to `"free_y"` instructs it to fix the width to the table width but keep the height free: p1 + wrap_table(aq, space = "free_y") ![](figs/unnamed-chunk-3-1.png) Setting `space` to `"fixed"` would constrain both the width and the height of the area it occupies. Since we only have a single row in our layout this would leave us with some empty horizontal space: p1 + wrap_table(aq, space = "fixed") ![](figs/unnamed-chunk-4-1.png) If the space is fixed in the y direction and the table has any source notes or footnotes, these will behave like the column header and be placed outside the panel area depending on the `panel` setting aq_footer <- gt(aq) |> tab_source_note("This is not part of the table body") p1 + wrap_table(aq_footer, space = "fixed") ![](figs/unnamed-chunk-5-1.png) While the space argument is great for making the composition look good and the table well placed in the whole, it can also serve a different purpose of making sure that rows (or columns) are aligned with the axis of a plot. There are no facilities to ensure that the breaks order matches between plots and tables so that is the responsibility of the user, but otherwise this is a great way to use tables to directly augment a plot: p2 <- ggplot(airquality) + geom_boxplot(aes(x = month.name[Month], y = Temp)) + theme(axis.text.x = element_blank(), axis.title.x = element_blank()) + scale_x_discrete(expand = c(0, 0.5)) # Construct our table table <- rbind( tapply(airquality$Temp, airquality$Month, max), tapply(airquality$Temp, airquality$Month, median), tapply(airquality$Temp, airquality$Month, min) ) colnames(table) <- month.name[5:9] table <- data.frame( Measure = c("Max", "Median", "Min"), table ) table <- gt(table, rowname_col = "Measure") |> cols_width(contains(month.name) ~ px(100)) |> cols_align(align = "center") |> cols_align(align = "right", columns = "Measure") p2 / wrap_table(table, space = "fixed") ![](figs/unnamed-chunk-6-1.png) Circling back, there was another argument to [`wrap_table()`](https://patchwork.data-imaginist.com/reference/wrap_table.html) we didn’t get into yet. In the plot above, we see that the row names are conveniently aligned with the axis rather than the panel of the plot above, in the same way as the headers where placed outside the panel area. This is a nice default and generally makes sense for the semantics of a table, but you might want something different. The `panel` argument allows you to control this exact behavior. It takes `"body"`, `"full"`, `"rows"`, or `"cols"` which indicate what portion of the table should be inside the panel area. The default is `"body"` which places row and column names outside the panel. `"full"`, on the contrary, places everything inside, while `"rows"` and `"cols"` are half versions that allows you to keep either column _or_ row names outside the panel respectively. # Place all rows (including the header row) inside the panel area p1 + wrap_table(aq, panel = "rows", space = "free_y") ![](figs/unnamed-chunk-7-1.png) Just like the tables support ggplot2-like titles, they also support tags, meaning that patchworks auto-tagging works as expected. It can be turned off using the `ignore_tag` argument but often you’d want to treat it as a figure in the figure text: p1 + wrap_table(aq, panel = "rows", space = "free_y") + plot_annotation(tag_levels = "A") & theme(plot.tag = element_text(margin = margin(0, 6, 6, 0))) ![](figs/unnamed-chunk-8-1.png) ### Accesibility[](#accesibility) We truly believe that the features laid out above will be a boon for augmenting your data visualisation with data that can be read precisely at a glance. However, we would be remiss to not note how tables that are part of a patchwork visualisation doesn’t have the same accessibility featurees as a gt table included directly in e.g. an HTML output. This is because graphics are rasterised into a PNG file and thus looses all semantical information that is inherent in a table. This should be kept in mind when providing Alt text for your figures so you ensure they are legible for everyone. ### Future[](#future) The support on the patchwork end is likely done at this point, but the conversion to grobs that has been added to gt is still somewhat young and will improve over time. It is likely that markdown formatting (through marquee) and other niceties will get added, leading to even more power in composing tables with plots using patchwork as the glue between them. As with the [support for gt in typst](https://quarto.org/docs/blog/posts/2024-07-02-beautiful-tables-in-typst/) the support for gt in patchwork is part of our larger effort to bring the power of gt to more environments and create a single unified solution to table styling. With freedom comes great responsibility[](#with-freedom-comes-great-responsibility) ------------------------------------------------------------------------------------ The second leg of this release concerns the [`free()`](https://patchwork.data-imaginist.com/reference/free.html) function which was introduced in the last release. I devoted a whole section of my posit::conf talk this year to talk about [`free()`](https://patchwork.data-imaginist.com/reference/free.html) and how it was a good thing to say no to requests for functionality until you have a solution that fits into your API and doesn’t add clutter. I really like how the API for [`free()`](https://patchwork.data-imaginist.com/reference/free.html) turned out but I also knew it could do more. In this release it delivers on those promises with two additional arguments. ### Which side?[](#which-side) As it were, [`free()`](https://patchwork.data-imaginist.com/reference/free.html) could only be used to completely turn off alignment of a plot, e.g. like below: p1 <- ggplot(mtcars) + geom_bar(aes(y = factor(gear), fill = factor(gear))) + scale_y_discrete( "", labels = c("3 gears are often enough", "But, you know, 4 is a nice number", "I would def go with 5 gears in a modern car") ) p2 <- ggplot(mtcars) + geom_point(aes(mpg, disp)) free(p1) / p2 ![](figs/unnamed-chunk-9-1.png) We can see that panel alignment has been turned off both to the left and to the right (and top and bottom if it were visible). But perhaps you are only interested in un-aligning the left side, keeping the legend to the right of both plots. Now you can, thanks to the `side` argument which takes a string containing one or more of the `t`, `r`, `b`, and `l` characters to indicate which sides to apply the freeing to (default is `"trbl"` meaning “target all sides”). free(p1, side = "l") / p2 ![](figs/unnamed-chunk-10-1.png) Freeing works inside nested patchworks, where you can target various sides at various levels: p3 <- ggplot(mtcars) + geom_boxplot(aes(y = factor(gear), disp)) + scale_y_discrete( "", labels = c("... and 3", "4 of them", "5 gears") ) nested <- p2 / free(p1, side = "l") free(nested, side = "r") / p3 ![](figs/unnamed-chunk-11-1.png) ### What does “freeing” means anyway?[](#what-does-freeing-means-anyway) While being able to target specific sides is pretty great in and off itself, we are not done yet. After being able to _not_ align panels the most requested feature was the possibility of moving the axis title closer to the axis text if alignment had pushed it apart. Consider again our unfreed patchwork: p1 / p2 ![](figs/unnamed-chunk-12-1.png) While we can “fix” it by letting the top panel stretch, another way to improve upon it would be to move the dangling y-axis title of the bottom plot closer to the axis. Enter the `type` argument to [`free()`](https://patchwork.data-imaginist.com/reference/free.html) which informs patchwork how to not align the input. The default (`"panel"`) works just as [`free()`](https://patchwork.data-imaginist.com/reference/free.html) always has, but the other two values opens up some new nifty goodies. Setting `type = "label"` does exactly what we discussed above, freeing the label from alignment so it sticks together with the axis and axis text: p1 / free(p2, type = "label") ![](figs/unnamed-chunk-13-1.png) The other type is `"space"` which works slightly different. Using this you tell patchwork to not reserve any space for what the side(s) contain. This is perfect in situation where you already have empty space next to it that can fit the content. Consider this plot: plot_spacer() + p1 + p2 + p2 ![](figs/unnamed-chunk-14-1.png) Ugh, the axis text of the top plot pushes everything apart even though there is ample of space for it in the empty region on the left. This is where `type = "space"` comes in handy: plot_spacer() + free(p1, type = "space", side = "l") + p2 + p2 ![](figs/unnamed-chunk-15-1.png) Of course, such power comes with the responsibility of you ensuring there is actually space for it — otherwise it will escape out of the figure area: free(p1, type = "space", side = "l") / p2 ![](figs/unnamed-chunk-16-1.png) All the different types of freeing can be stacked on top of each other so you can have a plot that keeps the left axis label together with the axis while also stretches the right side to take up empty space: p1 / free(free(p2, "panel", "r"), "label", "l") ![](figs/unnamed-chunk-17-1.png) But as always, don’t go overboard. If you find yourself needing to use an elaborate combination of stacked [`free()`](https://patchwork.data-imaginist.com/reference/free.html) calls there is a good chance that something with your core composition needs rethinking. The rest[](#the-rest) ---------------------- The above are the clear highlights of this release. It also contains the standard bug fixes — especially in the area of axis collecting which was introduced in the last release and came with a bunch of edge cases that were unaccounted for. There is also a new utility function: [`merge()`](https://rdrr.io/r/base/merge.html) which is an alternative to the `-` operator that I don’t think many users understood or used. It allows you to merge all plots together into a nested patchwork so that the right hand side is added to a new composition. Acknowledgements[](#acknowledgements) -------------------------------------- Thank you to all people who have contributed issues, code and comments to this release: [@BenVolpe94](https://github.com/BenVolpe94) , [@daniellembecker](https://github.com/daniellembecker) , [@dchiu911](https://github.com/dchiu911) , [@ericKuo722](https://github.com/ericKuo722) , [@Fan-iX](https://github.com/Fan-iX) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@jack-davison](https://github.com/jack-davison) , [@karchern](https://github.com/karchern) , [@laresbernardo](https://github.com/laresbernardo) , [@marchtaylor](https://github.com/marchtaylor) , [@mariadelmarq](https://github.com/mariadelmarq) , [@Maschette](https://github.com/Maschette) , [@michaeltopper1](https://github.com/michaeltopper1) , [@mkoohafkan](https://github.com/mkoohafkan) , [@n-kall](https://github.com/n-kall) , [@person-c](https://github.com/person-c) , [@pettyalex](https://github.com/pettyalex) , [@petzi53](https://github.com/petzi53) , [@phispu](https://github.com/phispu) , [@psychelzh](https://github.com/psychelzh) , [@rinivarg](https://github.com/rinivarg) , [@selkamand](https://github.com/selkamand) , [@Soham6298](https://github.com/Soham6298) , [@svraka](https://github.com/svraka) , [@teng-gao](https://github.com/teng-gao) , [@teunbrand](https://github.com/teunbrand) , [@thomasp85](https://github.com/thomasp85) , [@timz0605](https://github.com/timz0605) , [@wish1832](https://github.com/wish1832) , and [@Yunuuuu](https://github.com/Yunuuuu) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # WebAssembly roundup part 1: webR 0.4.2 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) WebAssembly roundup part 1: webR 0.4.2 ====================================== ![](/blog/2024/10/webr-0-4-2/thumbnail-wd.jpg) Photo by [Gatis Vilaks](https://unsplash.com/@gatisv)   2024/10/11   [webr](/tags/webr/) , [wasm](/tags/wasm/)   George Stagg We’re totally stoked to announce the release of [webr](https://docs.r-wasm.org/webr/v0.4.2/) 0.4.2! It’s been a little while since I’ve written about webR here, and a few releases between my last blog post and this one. In this post I’ll cover some of the exciting changes to the core webR distribution, and also include some interesting tidbits for JavaScript developers using webR in their own applications. You can see a full list of changes in the [release notes](https://github.com/r-wasm/webr/releases) . This post is the first in an R for WebAssembly roundup series. The next posts will cover updates to Shinylive for R, and introduce new a new Quarto extension that uses the power of webR and WebAssembly to elevate your documents with interactivity. * [WebAssembly roundup part 2: Shinylive 0.8.0](/blog/2024/10/shinylive-0-8-0/) * [WebAssembly roundup part 3: Quarto Live 0.1.1](/blog/2024/10/quarto-live-0-1-1/) Supporting HTML and widget display[](#supporting-html-and-widget-display) -------------------------------------------------------------------------- The base R distribution may be run using nothing but a text console, but some additional options can be implemented by frontends to provide system-dependent display of content. Previously, we implemented the [`pager`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/file.show.html) option so that R’s help system can be better displayed within the [webR application](https://webr.r-wasm.org/latest/) . Using the pager we can show R function and package documentation outside of the text console in dedicated tabbed windows. In recent releases of webR we have expanded our support for such display systems, providing an implementation both for the [`View()`](https://stat.ethz.ch/R-manual/R-devel/library/utils/html/View.html) function and the `viewer` global option used by [htmlwidgets](https://www.htmlwidgets.org/) . This gives us the ability to show a tabular data viewer for `data.frame`\-like R objects and an `iframe` based HTML content viewer, enabling dynamic web-based output from R packages like [`leaflet`](https://rstudio.github.io/leaflet/articles/leaflet.html) and [`gt`](https://gt.rstudio.com/) . ![Screenshots of the webR REPL showing a tabular data viewer, an interactive map using the leaflet package, and a HTML table rendered using the gt package.](images/viewer.png) The implementation of `viewer` is fairly general, making use of webR’s [output messages](https://docs.r-wasm.org/webr/latest/communication.html#output-messages) mechanism to send the required information to the main JavaScript thread for display. That way, any application using webR may choose to listen for those messages and how to show the resulting content on the page. We’ll make use of this in a later post where I introduce using webR to generate dynamic content in a new Quarto extension. Improvements to the webR app UI[](#improvements-to-the-webr-app-ui) -------------------------------------------------------------------- Recent webR releases have also made some other quality of life improvements to the webR app. Some minor improvements include making each UI panel resizeable, and offering `.zip` download of an entire directory in the Files panel. ![Screenshot showing the 'Download directory' feature in the webR REPL app.](images/download.png) ### R source syntax highlighting and parsing[](#r-source-syntax-highlighting-and-parsing) The webR app’s code editor is powered by [CodeMirror](https://codemirror.net/) with R parsing provided by the [codemirror-lang-r](https://github.com/TravisYeah/lang-r) package. CodeMirror’s extensibility is excellent, and the library is well suited for integrating into a wider project like this. However, we noticed that the `codemirror-lang-r` package had a few issues highlighting certain types of R syntax. In particular, in our application highlighting matrix operations such as `%*%` would crash the parser! ![Screenshots comparing syntax highlighting of R source code before and after the changes discussed above.](images/parser-crash.png) As well as fixing this bug, we’ve worked to improve the R parser to better support some other types of R syntax and have [contributed these changes upstream](https://github.com/TravisYeah/lezer-r/pull/1) so as to benefit other users of `codemirror-lang-r`. ![Screenshots comparing syntax highlighting of R source code before and after the changes discussed above.](images/parser-new.png) WebAssembly R package binary format[](#webassembly-r-package-binary-format) ---------------------------------------------------------------------------- One of R’s greatest strengths is its vibrant community of R packages and their developers, and so one of the development goals of webR is that packages are downloaded and installed as fast as possible. In the latest release of webR, some joint work with [Jeroen Ooms](https://github.com/jeroen) improving the performance of loading WebAssembly binary R packages has landed. R packages and other filesystem data is efficiently made available to the R WebAssembly process using [Emscripten’s file packager](https://emscripten.org/docs/porting/files/packaging_files.html#packaging-using-the-file-packager-tool) and the [`WORKERFS`](https://emscripten.org/docs/api_reference/Filesystem-API.html#filesystem-api-workerfs) filesystem driver. Previously we used uncompressed filesystem data, with the intention of serving content using [HTTP compression](https://en.wikipedia.org/wiki/HTTP_compression) . However, web services do not always compress files automatically[1](#fn:1) , especially if they are large. So, in the latest release of webR, filesystem data may now be mounted from a `gzip` compressed file[2](#fn:2) , and the base R filesystem is also distributed in compressed form. R package developers might recognise that traditional R package binaries are _already_ produced as a `gzip` compressed archive. And, as pointed out to me by Jeroen, the format of a `.tar` archive is very similar to Emscripten’s `.data` files. With some [clever arrangement](https://r-wasm.github.io/rwasm/articles/tar-metadata.html) of R package archive data and Emscripten filesystem metadata, pre-processed WebAssembly R package binaries may now be directly mounted to the virtual filesystem by webR. Mounting R packages in this way is more efficient than installing `.tgz` archives in the usual manner because the decompression step happens in the browser, rather than using R’s slower internal routines, and the `WORKERFS` filesystem driver also avoids memory copies with the archive files until they are actually opened and read by the WebAssembly R process. Both the [webR default repository](repo.r-wasm.org) and [R-Universe](https://r-universe.dev/) now serve binary R packages for WebAssembly in this new format. These packages can be installed and loaded interactively in the webR application, or used as dependencies in a deployed Shinylive for R app. For your own custom R packages, the [rwasm](https://r-wasm.github.io/rwasm/index.html) package can be used to compile WebAssembly binaries using a pre-configured Docker container. However, I’d actually recommend [creating a personal R-Universe repository](https://ropensci.org/blog/2021/06/22/setup-runiverse/) for your packages instead, since this will automatically build binaries for multiple targets including WebAssembly. A much simpler but effective change has also been made: R packages listed only as `LinkingTo` dependencies are no longer downloaded by webR on package installation. These are packages are required for building an R package from source, but _not at runtime_. The change saves network resources when installing WebAssembly R packages. In one particular worst-case scenario, this change avoided downloading about 100 megabytes of data! Virtual file system drivers[](#virtual-file-system-drivers) ------------------------------------------------------------ A nice side-effect of the work in the previous section is that [mounting filesystem data with `WORKERFS`](https://github.com/r-wasm/webr/issues/328) now also works correctly under Node.js, fixing a fairly painful and long-standing bug for our server-side users of webR. We’ve also introduced mounting with Emscripten’s [`IDBFS`](https://emscripten.org/docs/api_reference/Filesystem-API.html#filesystem-api-idbfs) filesystem driver when running webR in the browser[3](#fn:3) . This driver makes use of the low-level [IndexedDB API](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) provided by the JavaScript environment to write virtual filesystem contents to a form of local storage on the device. With this, files that have been written to the virtual filesystem can be persisted over page reloads and automatically made available again to the WebAssembly R process when the page is revisited in the future, without needing to re-download the content. You can try it out right here! Any files written to the `/persist` directory in the interactive R console below should be persisted. The first time you load this page, the directory will be empty. However, if files are written they will remain available after you refresh the page or revisit in the future. Loading webR... It should be noted that filesystem data stored in an IndexedDB database can only be accessed within the same [origin](https://developer.mozilla.org/en-US/docs/Glossary/Origin) , essentially across the current web page’s domain. Also, browsers may decide the amount of storage space provided, what content is deleted when quotas are reached, and when exactly that deletion occurs. In private browsing mode, for example, data is usually removed when the private session ends. Even with these caveats, I expect developers working with webR will be able to make use of the `IDBFS` driver to selectively cache content or R packages that are too large to download over the network on every single page load, further improving start up times in their own apps as a result. Developing with webR[](#developing-with-webr) ---------------------------------------------- ### Deprecating the `ServiceWorker` channel[](#deprecating-the-serviceworker-channel) The `ServiceWorker` [communication channel](https://docs.r-wasm.org/webr/v0.3.2/communication.html) , a method webR offered to handle message passing between the main browser thread and the [JavaScript Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) running the R WebAssembly binary, has been deprecated. The communication channel was originally devised as a way to allow use of webR in cases where the `SharedArrayBuffer` API is not available. This includes any use of webR with an origin that is not [Cross-Origin Isolated](https://developer.mozilla.org/en-US/docs/Web/API/Window/crossOriginIsolated) , such as when content is hosted by GitHub Pages. The channel was implemented using a [JavaScript Service Worker](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) proxy and synchronous [XHR](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) requests. Unfortunately, with the overhead of message serialisation and capturing network requests, performance was significantly impacted. The channel was also not compatible with applications that make use of a service worker for genuine network proxy functionality, such as Shinylive. An alternative method has since been developed in the form of the `PostMessage` communication channel. This instead uses the [JavaScript `PostMessage` API](https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage) , which is designed to handle communication between workers efficiently. It has much better performance and even provides a way to [transfer objects](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects) using zero-copy operations. There are some minor downsides when using the `PostMessage` channel, mostly related to taking input using tools like [`readline()`](https://rdrr.io/r/base/readline.html) , or nested REPLs like R’s [`browser()`](https://rdrr.io/r/base/browser.html) , but for most applications we find that this is not catastrophic and a reasonable price to pay for what is intended as a fallback method. If you are working on a webR application where [`readline()`](https://rdrr.io/r/base/readline.html) functionality _is absolutely_ required, but you cannot set your web server headers to enable cross-origin isolation, an alternative implementation of using a service worker to solve the problem can be found with the [coi-serviceworker](https://github.com/gzuidhof/coi-serviceworker) package. When enabled, the web page will appear to webR to be cross-origin isolated and so `SharedArrayBuffer` can be used. This still has the other drawbacks of requiring a service worker, but will have much better performance than using webR’s `ServiceWorker` channel directly. For these reasons, the `PostMessage` communication channel is now the default fallback when the web page is not cross-origin isolated. The `ServiceWorker` channel will continue to be available in the short-term, if explicitly requested, but will eventually be removed in a future version of webR. ### API additions[](#api-additions) We’ve made some minor changes to the webR JavaScript API. There’s nothing ground breaking here, but some new tools that we hope to be useful. ##### Report current version[](#report-current-version) With the aim of providing functionality similar to the [`R.Version()`](https://rdrr.io/r/base/Version.html) and [`packageVersion()`](https://rdrr.io/r/utils/packageDescription.html) R functions, the version of the currently running webR session may now be obtained from the JavaScript environment. > const webR = new WebR(); > webR.version; // '0.4.3-dev+d1fb4f4' ##### Discover an object’s class[](#discover-an-objects-class) An R object’s [`class()`](https://rdrr.io/r/base/class.html) may be inspected from an `RObject` proxy. The returned value is an `RCharacter` vector of classes from which the object inherits. > await webR.evalR("mtcars") .then(obj => obj.class()) .then(cls => cls.toArray()); // ['data.frame'] ##### Explicitly construct an R `data.frame`[](#explicitly-construct-an-r-dataframe) In a previous version of webR, we introduced creating new R `data.frame` objects from JavaScript using the generic `RObject` constructor. WebR will build a `data.frame` for arguments with compatible shape: either an object with named columns, or an array with objects for each row. > let source1 = { abc: [1, 2, 3], xyz: [4, 5, 6] }; > await new webR.RObject(source1) .then(obj => obj.class()) .then(cls => cls.toArray()); // ['data.frame'] > let source2 = [ { abc: 1, xyz: 4 }, { abc: 2, xyz: 5 }, { abc: 3, xyz: 6 }]; > await new webR.RObject(source2) .then(obj => obj.class()) .then(cls => cls.toArray()); // ['data.frame'] You might ask why not create an R list object by default? The reason is that we expect a common situation to be taking datasets defined in the JavaScript environment and processing them using R. With `data.frame` as the default, JavaScript objects that have been formatted for use with existing JavaScript frameworks can be almost transparently passed to R. > penguins; // Array(344) [\ // 0: { species: 'Adelie', island: 'Torgersen', flipper_length_mm: 181, ... }\ // ... more\ // ] > const sample_mass = await webR.evalR(` \\(x) x |> dplyr::sample_n(5) |> dplyr::pull("body_mass_g") `); > await sample_mass(penguins); // { type: 'double', names: null, values: [3300, 3250, 4000, 4700, 3750] } The generic constructor throws an exception for JavaScript objects that cannot be coerced as a `data.frame`. If you’d prefer to create an R list, you must instead be explicit by using the `RList` constructor, > let source = { def: [123, 456], uvw: 'hello' }; > await new webR.RObject(source); // Uncaught WebRWorkerError: Can't construct `data.frame`. Source object is not eligible. > let obj = await new webR.RList(source); > await obj.type(); // 'list' The `RObject` constructor is designed to be a useful default for interactive work at a JavaScript console. However, production applications should be [explicit in the choice of constructor](https://docs.r-wasm.org/webr/latest/api/js/modules/RWorker.html#classes) . With this in mind we have added a new class [`RDataFrame`](https://docs.r-wasm.org/webr/latest/api/js/classes/RWorker.RDataFrame.html) , a subclass of `RList`, so that users may be explicit in their choice of creating a `data.frame`, rather than relying on the generic `RObject` constructor. > let source = { abc: [1, 2, 3], xyz: [4, 5, 6] }; > await new webR.RDataFrame(source) .then(obj => obj.class()) .then(cls => cls.toArray()); // ['data.frame'] Now, if your source object is not quite as you expect, rather than continuing silently without error an exception will be thrown. We hope this will reduce the chance of type-related bugs and unexpected behaviour, and aid in debugging when issues do occur. // Say we _expect_ a JS object here, but something went wrong... > let bug = undefined; > const obj1 = await new webR.RObject(bug); // [No error and webR silently continues with an unexpected R object] > const obj2 = await new webR.RDataFrame(bug); // Uncaught WebRWorkerError: Can't construct `data.frame`. Source object is not eligible. Acknowledgements[](#acknowledgements) -------------------------------------- Special thanks to [@jeroen](https://github.com/jeroen) , for helpful conversations when it comes to packaging for webR. And thank you, as always, to the users and developers contributing to webR in the form of discussion, bug reports, and pull requests. [@027xiguapi](https://github.com/027xiguapi) , [@adrianolszewski](https://github.com/adrianolszewski) , [@alekrutkowski](https://github.com/alekrutkowski) , [@andrjohns](https://github.com/andrjohns) , [@baogorek](https://github.com/baogorek) , [@bugzpodder](https://github.com/bugzpodder) , [@christianp](https://github.com/christianp) , [@coatless](https://github.com/coatless) , [@codingthemystery](https://github.com/codingthemystery) , [@ColinFay](https://github.com/ColinFay) , [@derrickstaten](https://github.com/derrickstaten) , [@dipterix](https://github.com/dipterix) , [@EduardBel](https://github.com/EduardBel) , [@gregvolny](https://github.com/gregvolny) , [@guillaumechaumet](https://github.com/guillaumechaumet) , [@gyanaranjans](https://github.com/gyanaranjans) , [@helgasoft](https://github.com/helgasoft) , [@HenrikBengtsson](https://github.com/HenrikBengtsson) , [@isbool](https://github.com/isbool) , [@JosiahParry](https://github.com/JosiahParry) , [@luisDVA](https://github.com/luisDVA) , [@minhaj57sorder](https://github.com/minhaj57sorder) , [@olivroy](https://github.com/olivroy) , [@oranwutang](https://github.com/oranwutang) , [@pawelru](https://github.com/pawelru) , [@psychemedia](https://github.com/psychemedia) , [@rainer-rq-koelle](https://github.com/rainer-rq-koelle) , [@richarddmorey](https://github.com/richarddmorey) , [@richardjtelford](https://github.com/richardjtelford) , [@seanbirchall](https://github.com/seanbirchall) , [@shalom-lab](https://github.com/shalom-lab) , [@StaffanBetner](https://github.com/StaffanBetner) , [@stobor827](https://github.com/stobor827) , [@SugarRayLua](https://github.com/SugarRayLua) , [@tavosansal](https://github.com/tavosansal) , [@thomascwells](https://github.com/thomascwells) , [@timelyportfolio](https://github.com/timelyportfolio) , and [@zpinocchio](https://github.com/zpinocchio) . * * * 1. It depends a lot on how the hosting service has configured their production web server and the files themselves; both size and content type can make a difference to behaviour. Some services allow for pre-compressed content, while others do not. The [AWS CloudFront documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html) gives a good overview of how this all fits together. [↩︎](#fnref:1) 2. Emscripten’s `file_packager` tool also supports built-in `LZ4` compression with the `--lz4` flag. While generally useful for bundling files for WebAssembly applications, we avoid using this feature since it writes important data to a `.js` output file that must be executed. Ideally, we’d prefer our package loading mechanism to only require a single file download, similar to traditional R package archives. [↩︎](#fnref:2) 3. Note that currently users wanting to make use of `IDBFS` mounting must configure webR to use the [`PostMessage` Communication Channel](https://docs.r-wasm.org/webr/latest/communication.html) . [↩︎](#fnref:3) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Postprocessing is coming to tidymodels [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Postprocessing is coming to tidymodels ====================================== ![](/blog/2024/10/postprocessing-preview/thumbnail-wd.jpg) Photo by [Haley Owens](https://unsplash.com/photos/G6Y_YM3gO44)   2024/10/08   [tidymodels](/tags/tidymodels/) , [postprocessing](/tags/postprocessing/) , [workflows](/tags/workflows/)   Simon Couch, Hannah Frick, and Max Kuhn We’re bristling with elation to share about a set of upcoming features for postprocessing with tidymodels. Postprocessors refine predictions outputted from machine learning models to improve predictive performance or better satisfy distributional limitations. The developmental versions of many tidymodels core packages include changes to support postprocessors, and we’re ready to share about our work and hear the community’s thoughts on our progress so far. Postprocessing support with tidymodels hasn’t yet made it to CRAN, but you can install the needed versions of tidymodels packages with the following code. pak::pak( paste0( "tidymodels/", c("tune", "workflows", "rsample", "tailor") ) ) Now, we load packages with those developmental versions installed. library(tidymodels) library(probably) library(tailor) Existing tidymodels users might have spotted something funky already; who is this tailor character? Meet tailor👋[](#meet-tailor) ------------------------------ The tailor package introduces tailor objects, which compose iterative adjustments to model predictions. tailor is to postprocessing as recipes is to preprocessing; applying your mental model of recipes to tailor should get you a good bit of the way there. | Tool | Applied to... | Initialize with... | Composes... | Train with... | Predict with... | | --- | --- | --- | --- | --- | --- | | recipes | Training data | `recipe()` | `step_*()`s | `prep()` | `bake()` | | tailor | Model predictions | [`tailor()`](https://tailor.tidymodels.org/reference/tailor.html) | `adjust_*()`ments | [`fit()`](https://generics.r-lib.org/reference/fit.html) | [`predict()`](https://rdrr.io/r/stats/predict.html) | First, users can initialize a tailor object with [`tailor()`](https://tailor.tidymodels.org/reference/tailor.html) . tailor() #> #> ── tailor ────────────────────────────────────────────────────────────────────── #> A postprocessor with 0 adjustments. Tailors compose “adjustments,” analogous to steps from the recipes package. tailor() %>% adjust_probability_threshold(threshold = .7) #> #> ── tailor ────────────────────────────────────────────────────────────────────── #> A binary postprocessor with 1 adjustment: #> #> • Adjust probability threshold to 0.7. As an example, we’ll apply this tailor to the `two_class_example` data made available after loading tidymodels. head(two_class_example) #> truth Class1 Class2 predicted #> 1 Class2 0.003589243 0.9964107574 Class2 #> 2 Class1 0.678621054 0.3213789460 Class1 #> 3 Class2 0.110893522 0.8891064779 Class2 #> 4 Class1 0.735161703 0.2648382969 Class1 #> 5 Class2 0.016239960 0.9837600397 Class2 #> 6 Class1 0.999275071 0.0007249286 Class1 This data gives the true value of an outcome variable `truth` as well as predicted probabilities (`Class1` and `Class2`). The hard class predictions, in `predicted`, are `"Class1"` if the probability assigned to `"Class1"` is above .5, and `"Class2"` otherwise. The model predicts `"Class1"` more often than it does `"Class2"`. two_class_example %>% count(predicted) #> predicted n #> 1 Class1 277 #> 2 Class2 223 If we wanted the model to predict `"Class2"` more often, we could increase the probability threshold assigned to `"Class1"` above which the hard class prediction will be `"Class1"`. In the tailor package, this adjustment is implemented in [`adjust_probability_threshold()`](https://tailor.tidymodels.org/reference/adjust_probability_threshold.html) , which can be situated in a tailor object. tlr <- tailor() %>% adjust_probability_threshold(threshold = .7) tlr #> #> ── tailor ────────────────────────────────────────────────────────────────────── #> A binary postprocessor with 1 adjustment: #> #> • Adjust probability threshold to 0.7. tailors must be fitted before they can predict on new data. For adjustments like [`adjust_probability_threshold()`](https://tailor.tidymodels.org/reference/adjust_probability_threshold.html) , there’s no training that actually happens at the [`fit()`](https://generics.r-lib.org/reference/fit.html) step besides recording the name and type of relevant variables. For other adjustments, like numeric calibration with [`adjust_numeric_calibration()`](https://tailor.tidymodels.org/reference/adjust_numeric_calibration.html) , parameters are actually estimated at the [`fit()`](https://generics.r-lib.org/reference/fit.html) stage and separate data should be used to train the postprocessor and evaluate its performance. More on this in [Tailors in context](#tailors-in-context) . In this case, though, we can [`fit()`](https://generics.r-lib.org/reference/fit.html) on the whole dataset. The resulting object is still a tailor, but is now flagged as trained. tlr_trained <- fit( tlr, two_class_example, outcome = truth, estimate = predicted, probabilities = c(Class1, Class2) ) tlr_trained #> #> ── tailor ────────────────────────────────────────────────────────────────────── #> A binary postprocessor with 1 adjustment: #> #> • Adjust probability threshold to 0.7. [trained] When used with a model [workflow](https://workflows.tidymodels.org) via [`add_tailor()`](https://workflows.tidymodels.org/dev/reference/add_tailor.html) , the arguments to [`fit()`](https://generics.r-lib.org/reference/fit.html) a tailor will be set automatically. Generally, as in recipes, we recommend that users add tailors to model workflows for training and prediction rather than using them standalone for greater ease of use and to prevent data leakage, but tailors are totally functional by themselves, too. Now, when passed new data, the trained tailor will determine the outputted class based on whether the probability assigned to the level `"Class1"` is above `.7`, resulting in more predictions of `"Class2"` than before. predict(tlr_trained, two_class_example) %>% count(predicted) #> # A tibble: 2 × 2 #> predicted n #> #> 1 Class1 236 #> 2 Class2 264 Changing the probability threshold is one of many possible adjustments available in tailor. * For probabilities: [calibration](https://tailor.tidymodels.org/reference/adjust_probability_calibration.html) * For transformation of probabilities to hard class predictions: [thresholds](https://tailor.tidymodels.org/reference/adjust_probability_threshold.html) , [equivocal zones](https://tailor.tidymodels.org/reference/adjust_equivocal_zone.html) * For numeric outcomes: [calibration](https://tailor.tidymodels.org/reference/adjust_numeric_calibration.html) , [range](https://tailor.tidymodels.org/reference/adjust_numeric_range.html) Support for tailors is now plumbed through workflows (via [`add_tailor()`](https://workflows.tidymodels.org/dev/reference/add_tailor.html) ) and tune, and rsample includes a set of infrastructural changes to prevent data leakage behind the scenes. That said, we haven’t yet implemented support for tuning parameters in tailors, but we plan to implement that before this functionality heads to CRAN. Tailors in context[](#tailors-in-context) ------------------------------------------ As an example, let’s model a study of food delivery times in minutes (i.e., the time from the initial order to receiving the food) for a single restaurant. The `deliveries` data is available upon loading the tidymodels meta-package. data(deliveries) # split into training and testing sets set.seed(1) delivery_split <- initial_split(deliveries) delivery_train <- training(delivery_split) delivery_test <- testing(delivery_split) # resample the training set using 10-fold cross-validation set.seed(1) delivery_folds <- vfold_cv(delivery_train) # print out the training set delivery_train #> # A tibble: 7,509 × 31 #> time_to_delivery hour day distance item_01 item_02 item_03 item_04 item_05 #> #> 1 21.2 16.1 Tue 3.02 0 0 0 0 0 #> 2 17.9 12.4 Sun 3.37 0 0 0 0 0 #> 3 22.4 14.2 Fri 2.59 0 0 0 0 0 #> 4 30.9 19.1 Sat 2.77 0 0 0 0 0 #> 5 30.1 16.5 Fri 2.05 0 0 0 1 0 #> 6 35.3 14.7 Sat 4.57 0 0 2 1 1 #> 7 13.1 11.5 Sat 2.09 0 0 0 0 0 #> 8 18.3 13.4 Tue 2.35 0 2 1 0 0 #> 9 25.2 20.5 Sat 2.43 0 0 0 1 0 #> 10 30.7 16.7 Fri 2.24 0 0 0 1 0 #> # ℹ 7,499 more rows #> # ℹ 22 more variables: item_06 , item_07 , item_08 , #> # item_09 , item_10 , item_11 , item_12 , item_13 , #> # item_14 , item_15 , item_16 , item_17 , item_18 , #> # item_19 , item_20 , item_21 , item_22 , item_23 , #> # item_24 , item_25 , item_26 , item_27 Let’s deliberately define a regression model that has poor predicted values: a boosted tree with only three ensemble members. delivery_wflow <- workflow() %>% add_formula(time_to_delivery ~ .) %>% add_model(boost_tree(mode = "regression", trees = 3)) Evaluating against resamples: set.seed(1) delivery_res <- fit_resamples( delivery_wflow, delivery_folds, control = control_resamples(save_pred = TRUE) ) The $R^2$ looks quite strong! collect_metrics(delivery_res) #> # A tibble: 2 × 6 #> .metric .estimator mean n std_err .config #> #> 1 rmse standard 9.52 10 0.0533 Preprocessor1_Model1 #> 2 rsq standard 0.853 10 0.00357 Preprocessor1_Model1 Let’s take a closer look at the predictions, though. How well are they calibrated? We can use the [`cal_plot_regression()`](https://probably.tidymodels.org/reference/cal_plot_regression.html) helper from the probably package to put together a quick diagnostic plot. collect_predictions(delivery_res) %>% cal_plot_regression(truth = time_to_delivery, estimate = .pred) ![](figs/predictions-bad-boost-1.png) Ooof. In comes tailor! Numeric calibration can help address the correlated errors here. We can add a tailor to our existing workflow to “bump up” predictions towards their true value. delivery_wflow_improved <- delivery_wflow %>% add_tailor(tailor() %>% adjust_numeric_calibration()) The resampling code looks the same from here. set.seed(1) delivery_res_improved <- fit_resamples( delivery_wflow_improved, delivery_folds, control = control_resamples(save_pred = TRUE) ) Checking out the same plot reveals a much better fit! collect_predictions(delivery_res_improved) %>% cal_plot_regression(truth = time_to_delivery, estimate = .pred) ![](figs/predictios-better-boost-1.png) There’s actually some tricky data leakage prevention happening under the hood here. When you add tailors to workflow and fit them with tune, this is all taken care of for you. If you’re interested in using tailors outside of that context, check out [this documentation section](https://workflows.tidymodels.org/dev/reference/add_tailor.html#data-usage) in `add_tailor()`. What’s to come[](#whats-to-come) --------------------------------- We’re excited about how this work is shaping up and would love to hear yall’s thoughts on what we’ve brought together so far. Please do comment on our social media posts about this blog entry or leave issues on the [tailor GitHub repository](https://github.com/tidymodels/tailor) and let us know what you think! Before these changes head out to CRAN, we’ll also be implementing tuning functionality for postprocessors. You’ll be able to tag arguments like `adjust_probability_threshold(threshold)` or `adjust_probability_calibration(method)` with `tune()` to optimize across several values. Besides that, post-processing with tidymodels should “just work” on the developmental versions of our packages—let us know if you come across anything wonky. Acknowledgements[](#acknowledgements) -------------------------------------- Postprocessing support has been a longstanding feature request across many of our repositories; we’re grateful for the community discussions there for shaping this work. Additionally, we thank Ryan Tibshirani and Daniel McDonald for fruitful discussions on how we might scope these features. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # devtools [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Devtools [![](/blog/2024/07/pkgdown-2-1-0/thumbnail-sq.jpg)](/blog/2024/07/pkgdown-2-1-0/) [pkgdown 2.1.0](/blog/2024/07/pkgdown-2-1-0/) [package](/categories/package) Hadley Wickham pkgdown 2.1.0 includes two major new features: support for quarto vignettes and a “light switch” that lets the reader switch between light and dark mode. It also contains a bunch of other improvements to both the user and the developer experience. [Read more ...](/blog/2024/07/pkgdown-2-1-0/) 2024/07/08 [![](/blog/2023/10/testthat-3-2-0/thumbnail-sq.jpg)](/blog/2023/10/testthat-3-2-0/) [testthat 3.2.0](/blog/2023/10/testthat-3-2-0/) [package](/categories/package) Hadley Wickham Catch up on the last two years of testthat development which includes improved documentation, new expectations, a new style for error snapshots, support for mocking, a new way to detect if a test has changed global state, and a handful of UI improvements. [Read more ...](/blog/2023/10/testthat-3-2-0/) 2023/10/08 [![](/blog/2022/05/roxygen2-7-2-0/thumbnail-sq.jpg)](/blog/2022/05/roxygen2-7-2-0/) [roxygen2 7.2.0](/blog/2022/05/roxygen2-7-2-0/) [package](/categories/package) Hadley Wickham roxygen2 7.2.0 brings improvements to `NAMESPACE` generation, better multiparameter argument inheritance, and improved warnings. [Read more ...](/blog/2022/05/roxygen2-7-2-0/) 2022/05/13 [![](/blog/2022/02/upkeep-testthat-3/thumbnail-sq.jpg)](/blog/2022/02/upkeep-testthat-3/) [Upgrading to testthat edition 3](/blog/2022/02/upkeep-testthat-3/) [learn](/categories/learn) Hannah Frick A workflow for upgrading to testthat edition 3: activation, deprecations, changes to warnings, messages, and comparisons. [Read more ...](/blog/2022/02/upkeep-testthat-3/) 2022/02/22 [![](/blog/2021/12/pkgdown-2-0-0/thumbnail-sq.jpg)](/blog/2021/12/pkgdown-2-0-0/) [pkgdown 2.0.0](/blog/2021/12/pkgdown-2-0-0/) [package](/categories/package) Hadley Wickham pkgdown 2.0.0 includes a major refresh of the default template (now using bootstrap 5), many new ways to customise your site, improvements to code styling, and much, much, more. [Read more ...](/blog/2021/12/pkgdown-2-0-0/) 2021/12/03 [![](/blog/2021/10/renaming-default-branch/thumbnail-sq.jpg)](/blog/2021/10/renaming-default-branch/) [Renaming the default branch](/blog/2021/10/renaming-default-branch/) [learn](/categories/learn) [package](/categories/package) Jenny Bryan We are renaming the default branch of many Git(Hub) repositories and this post explains how contributors can adapt, using new functionality in usethis. [Read more ...](/blog/2021/10/renaming-default-branch/) 2021/10/27 [![](/blog/2020/12/usethis-2-0-0/thumbnail-sq.jpg)](/blog/2020/12/usethis-2-0-0/) [usethis 2.0.0](/blog/2020/12/usethis-2-0-0/) [package](/categories/package) Jenny Bryan This is a big release aimed at improving usability, especially around Git and GitHub functionality. [Read more ...](/blog/2020/12/usethis-2-0-0/) 2020/12/10 [![](/blog/2020/04/usethis-1-6-0/thumbnail-sq.jpg)](/blog/2020/04/usethis-1-6-0/) [usethis 1.6.0](/blog/2020/04/usethis-1-6-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2020/04/usethis-1-6-0/) 2020/04/11 [![](/blog/2020/03/roxygen2-7-1-0/thumb-sq.jpg)](/blog/2020/03/roxygen2-7-1-0/) [roxygen2 7.1.0](/blog/2020/03/roxygen2-7-1-0/) [package](/categories/package) Gábor Csárdi A minor roxygen2 release with some new features [Read more ...](/blog/2020/03/roxygen2-7-1-0/) 2020/03/11 [![](/blog/2019/11/roxygen2-7-0-0/thumb-sq.jpg)](/blog/2019/11/roxygen2-7-0-0/) [roxygen2 7.0.0](/blog/2019/11/roxygen2-7-0-0/) [package](/categories/package) Hadley Wickham A massive update to roxygen2 now on CRAN. [Read more ...](/blog/2019/11/roxygen2-7-0-0/) 2019/11/12 * [««](/tags/devtools/) * « * [1](/tags/devtools/) * [2](/tags/devtools/page/2/) * [»](/tags/devtools/page/2/) * [»»](/tags/devtools/page/2/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # recipes [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Recipes [![](/blog/2024/07/recipes-1-1-0/thumbnail-sq.jpg)](/blog/2024/07/recipes-1-1-0/) [recipes 1.1.0](/blog/2024/07/recipes-1-1-0/) [package](/categories/package) Emil Hvitfeldt recipes 1.1.0 is on CRAN! recipes now have better input checking and quality of life errors. [Read more ...](/blog/2024/07/recipes-1-1-0/) 2024/07/08 [![](/blog/2024/01/tidymodels-2023-q4/thumbnail-sq.jpg)](/blog/2024/01/tidymodels-2023-q4/) [Q4 2023 tidymodels digest](/blog/2024/01/tidymodels-2023-q4/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2024/01/tidymodels-2023-q4/) 2024/01/09 [![](/blog/2023/04/tidymodels-2023-q1/thumbnail-sq.jpg)](/blog/2023/04/tidymodels-2023-q1/) [Q1 2023 tidymodels digest](/blog/2023/04/tidymodels-2023-q1/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/04/tidymodels-2023-q1/) 2023/04/28 [![](/blog/2022/12/tidymodels-2022-q4/thumbnail-sq.jpg)](/blog/2022/12/tidymodels-2022-q4/) [Q4 2022 tidymodels digest](/blog/2022/12/tidymodels-2022-q4/) [roundup](/categories/roundup) Simon Couch The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2022/12/tidymodels-2022-q4/) 2022/12/29 [![](/blog/2022/10/tidymodels-2022-q3/thumbnail-sq.jpg)](/blog/2022/10/tidymodels-2022-q3/) [Q3 2022 tidymodels digest](/blog/2022/10/tidymodels-2022-q3/) [roundup](/categories/roundup) Max Kuhn Our post-RStudio conference productivity has been high! This post talks about tidymodels updates from the last few months. [Read more ...](/blog/2022/10/tidymodels-2022-q3/) 2022/10/19 [![](/blog/2022/07/tidymodels-2022-q2/thumbnail-sq.jpg)](/blog/2022/07/tidymodels-2022-q2/) [Q2 2022 tidymodels digest](/blog/2022/07/tidymodels-2022-q2/) [roundup](/categories/roundup) Emil Hvitfeldt Q2 marks the end of the season of case weights, with 25 new releases. [Read more ...](/blog/2022/07/tidymodels-2022-q2/) 2022/07/19 [![](/blog/2022/05/recipes-update-05-20222/thumbnail-sq.jpg)](/blog/2022/05/recipes-update-05-20222/) [Updates for recipes extension packages](/blog/2022/05/recipes-update-05-20222/) [package](/categories/package) Emil Hvitfeldt The three extension packages for recipes were recently updated on CRAN adding new steps, features and bug fixes. [Read more ...](/blog/2022/05/recipes-update-05-20222/) 2022/05/03 [![](/blog/2022/02/recipes-0-2-0/thumbnail-sq.jpg)](/blog/2022/02/recipes-0-2-0/) [recipes 0.2.0](/blog/2022/02/recipes-0-2-0/) [package](/categories/package) Max Kuhn Recipes has added a few new steps along with many improvements. [Read more ...](/blog/2022/02/recipes-0-2-0/) 2022/02/22 [![](/blog/2021/09/tidymodels-2021-q3/thumbnail-sq.jpg)](/blog/2021/09/tidymodels-2021-q3/) [Q3 2021 tidymodels roundup](/blog/2021/09/tidymodels-2021-q3/) [roundup](/categories/roundup) Julia Silge Use new tuning parameters, new recipe steps, and a new example dataset! [Read more ...](/blog/2021/09/tidymodels-2021-q3/) 2021/09/28 [![](/blog/2021/07/tidymodels-2021-q2/thumbnail-sq.jpg)](/blog/2021/07/tidymodels-2021-q2/) [Q2 2021 tidymodels digest](/blog/2021/07/tidymodels-2021-q2/) [roundup](/categories/roundup) Julia Silge Releases of tidymodels packages in Q2 of 2021 include more streamlined memory footprints for feature-engineering recipes, new model engine options, and better support for post-processing predictions. [Read more ...](/blog/2021/07/tidymodels-2021-q2/) 2021/07/02 * [««](/tags/recipes/) * « * [1](/tags/recipes/) * [2](/tags/recipes/page/2/) * [»](/tags/recipes/page/2/) * [»»](/tags/recipes/page/2/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # pkgdown 2.1.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) pkgdown 2.1.0 ============= ![](/blog/2024/07/pkgdown-2-1-0/thumbnail-wd.jpg) Photo by [ChatGPT 4o](https://chatgpt.com/)   2024/07/08   [devtools](/tags/devtools/) , [pkgdown](/tags/pkgdown/)   Hadley Wickham We’re delighted to announce the release of [pkgdown](http://pkgdown.r-lib.org/) 2.1.0. pkgdown is designed to make it quick and easy to build a beautiful and accessible website for your package. You can install it from CRAN with: install.packages("pkgdown") This is a massive release with a bunch of new features. I’ll highlight the most important here, but as always, I highlight recommend skimming the [release notes](https://github.com/r-lib/pkgdown/releases/tag/v2.1.0) for other smaller improvements and bug fixes. First, and most importantly, please join me in welcoming two new authors to pkgdown: [Olivier Roy](https://github.com/olivroy) and [Salim Brüggemann](https://github.com/salim-b) . They have both contributed many improvements to the package and I’m very happy to officially have them aboard as package authors. library(pkgdown) Lifecycle changes[](#lifecycle-changes) ---------------------------------------- Let’s get started with the important stuff, the [lifecycle updates](https://www.tidyverse.org/blog/2021/02/lifecycle-1-0-0/) . Most important we’ve decided to deprecate support for Bootstrap 3, which was superseded in December 2021. We’re starting to more directly encourage folks to move away from it as maintaining two separate sets of site templates is a time sink. If you’re still using BS3, now’s the [time to upgrade](https://www.tidyverse.org/blog/2021/12/pkgdown-2-0-0/#bootstrap-5) . There are three other changes that are less likely to affect folks: * The `document` argument to [`build_site()`](https://pkgdown.r-lib.org/reference/build_site.html) and [`build_reference()`](https://pkgdown.r-lib.org/reference/build_reference.html) has been removed after being deprecated in pkgdown 1.4.0; use the [`devel` argument](https://pkgdown.r-lib.org/reference/build_site.html#arg-devel) instead. * [`autolink_html()`](https://pkgdown.r-lib.org/reference/autolink_html.html) was deprecated in pkgdown 1.6.0 and now warns every time you use it; use [`downlit::downlit_html_path()`](https://downlit.r-lib.org/reference/downlit_html_path.html) instead. * [`preview_page()`](https://pkgdown.r-lib.org/reference/preview_page.html) has been deprecated; use [`preview_site()`](https://pkgdown.r-lib.org/reference/preview_site.html) instead. Major new features[](#major-new-features) ------------------------------------------ pkgdown 2.1.0 has two major new features: support for Quarto vignettes and a new light switch that toggles between light and dark modes. ### Quarto support[](#quarto-support) [`build_article()`](https://pkgdown.r-lib.org/reference/build_articles.html) / [`build_articles()`](https://pkgdown.r-lib.org/reference/build_articles.html) now support articles and vignettes written with Quarto. To use it, make sure you have the the latest version of Quarto, 1.5, which was released last week. By and large you should be able to just write in Quarto and things will just work, but you will need to make a small change to your GitHub action. Learn more at [`vignette("quarto")`](https://pkgdown.r-lib.org/articles/quarto.html) . Combining the individual quarto and pkgdown templating systems is a delicate art, so while I’ve done my best to make it work, there may be some rough edges. Check out the current known limitations in [`vignette("quarto")`](https://pkgdown.r-lib.org/articles/quarto.html) , and please file an issue if you encounter a quarto feature that doesn’t work quite right. ### Light switch[](#light-switch) pkgdown sites can now provide a “light switch” that allows the reader to switch between light and dark modes (based on work in bslib by [@gadenbuie](https://github.com/gadenbuie) ). You can try it out on [https://pkgdown.r-lib.org](https://pkgdown.r-lib.org) : the light switch appears at the far right at the navbar and remembers the users choice between visits to your site. (Note that the light switch works differently to quarto dark mode. In quarto, you can provide two completely different themes for light and dark mode. In pkgdown, dark mode is a relatively thin overlay that based on your light theme colours.) For now, you’ll need to opt-in to the light-switch by adding the following to your `_pkgdown.yml`: template light-switch: true In the future we hope to turn it on automatically. You can learn more about customising the light switch in [`vignette("customise")`](https://pkgdown.r-lib.org/articles/customise.html) : you can choose to select your own syntax highlighting scheme for dark mode, override dark-specific BS lib variables, and move its location in the navbar. User experience[](#user-experience) ------------------------------------ We’ve made a bunch of small changes to enhance the user experience of pkgdown sites: * We’ve continued in our efforts to make pkgdown sites as accessible as possible by now warning if you’ve forgotten to add alt text to images (including plots) in your articles. We’ve also added a new [`vignette("accessibility")`](https://pkgdown.r-lib.org/articles/accessibility.html) which describes additional manual tasks you can perform to make your site as accessible as possible. * [`build_reference()`](https://pkgdown.r-lib.org/reference/build_reference.html) adds anchors to arguments making it possible to link directly to an argument. This is very useful when you’re trying to direct folks to the documentation for a specific argument, e.g. [https://pkgdown.r-lib.org/reference/build\_site.html#arg-devel](https://pkgdown.r-lib.org/reference/build_site.html#arg-devel) . * [`build_reference_index()`](https://pkgdown.r-lib.org/reference/build_reference.html) now displays function lifecycle badges [next to the function name](https://pkgdown.r-lib.org/reference/index.html#deprecated-functions) . If you want to gather together (e.g.) all the deprecated function in one spot in the reference index, you can use the new topic selector `has_lifecycle("deprecated")`. * The new `template.math-rendering` option allows you to control how math is rendered on your site. The default uses `mathml` which is zero dependency but has the lowest fidelity. If you use a lot of math on your site, you can switch back to the previous method with `mathjax`, or try out `katex`, a faster alternative. * pkgdown sites no longer depend on external content distribution networks (CDN) for common javascript, CSS, and font files. CDNs no longer provide [any performance advantages](https://www.stefanjudis.com/notes/say-goodbye-to-resource-caching-across-sites-and-domains/) and make deployment harder inside certain locked-down corporate environments. * pkgdown includes translations for more terms including “Abstract” and “Search site”. A big thanks to @jplecavalier, @dieghernan, @krlmlr, @LDalby, @rich-iannone, @jmaspons, and @mine-cetinkaya-rundel for providing updated translations in French, Spanish, Portugese, Germna, Catalan, and Turkish! I’ve also written [`vignette("translations")`](https://pkgdown.r-lib.org/articles/translations.html) , a brief vignette that discusses how translation works for non-English sites, and includes how you can create translations for new languages. (This is a great way to contribute to pkgdown if you are multi-lingual!) ### Developer experience[](#developer-experience) We’ve also made a bunch of minor improvements to make improve the package developer experience: * YAML validation has been substantially improved so you should get much clearer errors if you have made a mistake in your `_pkgdown.yml`. Please [file an issue](https://github.com/r-lib/pkgdown/issues/new) if you find a case where the error message is not helpful. * The `build_*()` functions (apart from [`build_site()`](https://pkgdown.r-lib.org/reference/build_site.html) ) no longer automatically preview in interactive sessions since they all emit clickable links to any files that have changed. You can continue to use [`preview_site()`](https://pkgdown.r-lib.org/reference/preview_site.html) to open the site in your browser. * The `build_*()` functions now work better if you’re previewing just part of a site and haven’t built the whole thing. It should no longer be necessary to run [`init_site()`](https://pkgdown.r-lib.org/reference/init_site.html) in most cases, and you shouldn’t be able to get into a state where you’re told to run [`init_site()`](https://pkgdown.r-lib.org/reference/init_site.html) and then it doesn’t work. * We give more and clearer details of the site building process including reporting on exactly what is generated by bslib, what is copied from templates, and what redirects are generated. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 212 folks who contributed to this release! [@Adafede](https://github.com/Adafede) , [@AEBilgrau](https://github.com/AEBilgrau) , [@albertocasagrande](https://github.com/albertocasagrande) , [@alex-d13](https://github.com/alex-d13) , [@AliSajid](https://github.com/AliSajid) , [@arkadiuszbeer](https://github.com/arkadiuszbeer) , [@ArneBab](https://github.com/ArneBab) , [@asadow](https://github.com/asadow) , [@ateucher](https://github.com/ateucher) , [@avhz](https://github.com/avhz) , [@banfai](https://github.com/banfai) , [@barcaroli](https://github.com/barcaroli) , [@BartJanvanRossum](https://github.com/BartJanvanRossum) , [@bastistician](https://github.com/bastistician) , [@ben18785](https://github.com/ben18785) , [@bijoychandraAU](https://github.com/bijoychandraAU) , [@Bisaloo](https://github.com/Bisaloo) , [@bkmgit](https://github.com/bkmgit) , [@bnprks](https://github.com/bnprks) , [@brycefrank](https://github.com/brycefrank) , [@bschilder](https://github.com/bschilder) , [@bundfussr](https://github.com/bundfussr) , [@cararthompson](https://github.com/cararthompson) , [@Carol-seven](https://github.com/Carol-seven) , [@cbailiss](https://github.com/cbailiss) , [@cboettig](https://github.com/cboettig) , [@cderv](https://github.com/cderv) , [@chlebowa](https://github.com/chlebowa) , [@chuxinyuan](https://github.com/chuxinyuan) , [@cromanpa94](https://github.com/cromanpa94) , [@cthombor](https://github.com/cthombor) , [@d-morrison](https://github.com/d-morrison) , [@DanChaltiel](https://github.com/DanChaltiel) , [@DarioS](https://github.com/DarioS) , [@davidchall](https://github.com/davidchall) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dbosak01](https://github.com/dbosak01) , [@dchiu911](https://github.com/dchiu911) , [@ddsjoberg](https://github.com/ddsjoberg) , [@DeepanshKhurana](https://github.com/DeepanshKhurana) , [@dhersz](https://github.com/dhersz) , [@dieghernan](https://github.com/dieghernan) , [@djhocking](https://github.com/djhocking) , [@dkarletsos](https://github.com/dkarletsos) , [@dmurdoch](https://github.com/dmurdoch) , [@dshemetov](https://github.com/dshemetov) , [@dsweber2](https://github.com/dsweber2) , [@dvg-p4](https://github.com/dvg-p4) , [@DyfanJones](https://github.com/DyfanJones) , [@ecmerkle](https://github.com/ecmerkle) , [@eddelbuettel](https://github.com/eddelbuettel) , [@eeholmes](https://github.com/eeholmes) , [@eitsupi](https://github.com/eitsupi) , [@eliocamp](https://github.com/eliocamp) , [@elong0527](https://github.com/elong0527) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@erikarasnick](https://github.com/erikarasnick) , [@esimms999](https://github.com/esimms999) , [@espinielli](https://github.com/espinielli) , [@etiennebacher](https://github.com/etiennebacher) , [@ewenharrison](https://github.com/ewenharrison) , [@filipsch](https://github.com/filipsch) , [@FlukeAndFeather](https://github.com/FlukeAndFeather) , [@francoisluc](https://github.com/francoisluc) , [@friendly](https://github.com/friendly) , [@fweber144](https://github.com/fweber144) , [@gaborcsardi](https://github.com/gaborcsardi) , [@gadenbuie](https://github.com/gadenbuie) , [@galachad](https://github.com/galachad) , [@gangstR](https://github.com/gangstR) , [@gavinsimpson](https://github.com/gavinsimpson) , [@GeoBosh](https://github.com/GeoBosh) , [@GFabien](https://github.com/GFabien) , [@ggcostoya](https://github.com/ggcostoya) , [@ghost](https://github.com/ghost) , [@givison](https://github.com/givison) , [@gladkia](https://github.com/gladkia) , [@glin](https://github.com/glin) , [@gmbecker](https://github.com/gmbecker) , [@gravesti](https://github.com/gravesti) , [@GregorDeCillia](https://github.com/GregorDeCillia) , [@gregorypenn](https://github.com/gregorypenn) , [@gsmolinski](https://github.com/gsmolinski) , [@gsrohde](https://github.com/gsrohde) , [@gungorMetehan](https://github.com/gungorMetehan) , [@hadley](https://github.com/hadley) , [@harshkrishna17](https://github.com/harshkrishna17) , [@HenrikBengtsson](https://github.com/HenrikBengtsson) , [@hfrick](https://github.com/hfrick) , [@hrecht](https://github.com/hrecht) , [@hsloot](https://github.com/hsloot) , [@idavydov](https://github.com/idavydov) , [@idmn](https://github.com/idmn) , [@igordot](https://github.com/igordot) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@jabenninghoff](https://github.com/jabenninghoff) , [@jack-davison](https://github.com/jack-davison) , [@jangorecki](https://github.com/jangorecki) , [@jayhesselberth](https://github.com/jayhesselberth) , [@jennybc](https://github.com/jennybc) , [@jeroen](https://github.com/jeroen) , [@JerryWho](https://github.com/JerryWho) , [@jhelvy](https://github.com/jhelvy) , [@jmaspons](https://github.com/jmaspons) , [@john-harrold](https://github.com/john-harrold) , [@john-ioannides](https://github.com/john-ioannides) , [@jonasmuench](https://github.com/jonasmuench) , [@jonnybaik](https://github.com/jonnybaik) , [@josherrickson](https://github.com/josherrickson) , [@joshualerickson](https://github.com/joshualerickson) , [@JosiahParry](https://github.com/JosiahParry) , [@jplecavalier](https://github.com/jplecavalier) , [@JSchoenbachler](https://github.com/JSchoenbachler) , [@juliasilge](https://github.com/juliasilge) , [@jwimberl](https://github.com/jwimberl) , [@kalaschnik](https://github.com/kalaschnik) , [@kevinushey](https://github.com/kevinushey) , [@klmr](https://github.com/klmr) , [@krlmlr](https://github.com/krlmlr) , [@LDalby](https://github.com/LDalby) , [@ldecicco-USGS](https://github.com/ldecicco-USGS) , [@lhdjung](https://github.com/lhdjung) , [@LiNk-NY](https://github.com/LiNk-NY) , [@lionel-](https://github.com/lionel-) , [@Liripo](https://github.com/Liripo) , [@lorenzwalthert](https://github.com/lorenzwalthert) , [@lschneiderbauer](https://github.com/lschneiderbauer) , [@mabesa](https://github.com/mabesa) , [@maelle](https://github.com/maelle) , [@maRce10](https://github.com/maRce10) , [@margotbligh](https://github.com/margotbligh) , [@marine-ecologist](https://github.com/marine-ecologist) , [@markfairbanks](https://github.com/markfairbanks) , [@martinlaw](https://github.com/martinlaw) , [@matt-dray](https://github.com/matt-dray) , [@mattfidler](https://github.com/mattfidler) , [@matthewjnield](https://github.com/matthewjnield) , [@MattPM](https://github.com/MattPM) , [@mccarthy-m-g](https://github.com/mccarthy-m-g) , [@MEO265](https://github.com/MEO265) , [@merliseclyde](https://github.com/merliseclyde) , [@MichaelChirico](https://github.com/MichaelChirico) , [@mikeblazanin](https://github.com/mikeblazanin) , [@mikeroswell](https://github.com/mikeroswell) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@MLopez-Ibanez](https://github.com/MLopez-Ibanez) , [@Moohan](https://github.com/Moohan) , [@mpadge](https://github.com/mpadge) , [@mrcaseb](https://github.com/mrcaseb) , [@mrchypark](https://github.com/mrchypark) , [@ms609](https://github.com/ms609) , [@msberends](https://github.com/msberends) , [@musvaage](https://github.com/musvaage) , [@nanxstats](https://github.com/nanxstats) , [@nathaneastwood](https://github.com/nathaneastwood) , [@netique](https://github.com/netique) , [@nicholascarey](https://github.com/nicholascarey) , [@nicolerg](https://github.com/nicolerg) , [@olivroy](https://github.com/olivroy) , [@pearsonca](https://github.com/pearsonca) , [@peterdesmet](https://github.com/peterdesmet) , [@phauchamps](https://github.com/phauchamps) , [@przmv](https://github.com/przmv) , [@quantsch](https://github.com/quantsch) , [@ramiromagno](https://github.com/ramiromagno) , [@rcannood](https://github.com/rcannood) , [@rempsyc](https://github.com/rempsyc) , [@rgaiacs](https://github.com/rgaiacs) , [@rich-iannone](https://github.com/rich-iannone) , [@rickhelmus](https://github.com/rickhelmus) , [@rmflight](https://github.com/rmflight) , [@robmoss](https://github.com/robmoss) , [@royfrancis](https://github.com/royfrancis) , [@rsangole](https://github.com/rsangole) , [@ryantibs](https://github.com/ryantibs) , [@salim-b](https://github.com/salim-b) , [@samuel-marsh](https://github.com/samuel-marsh) , [@SebKrantz](https://github.com/SebKrantz) , [@SESjo](https://github.com/SESjo) , [@sgvignali](https://github.com/sgvignali) , [@spsanderson](https://github.com/spsanderson) , [@srfall](https://github.com/srfall) , [@stefanoborini](https://github.com/stefanoborini) , [@stephenashton-dhsc](https://github.com/stephenashton-dhsc) , [@strengejacke](https://github.com/strengejacke) , [@swsoyee](https://github.com/swsoyee) , [@t-kalinowski](https://github.com/t-kalinowski) , [@talgalili](https://github.com/talgalili) , [@tanho63](https://github.com/tanho63) , [@tedmoorman](https://github.com/tedmoorman) , [@telphick](https://github.com/telphick) , [@TFKentUSDA](https://github.com/TFKentUSDA) , [@ThierryO](https://github.com/ThierryO) , [@thisisnic](https://github.com/thisisnic) , [@thomasp85](https://github.com/thomasp85) , [@tomsing1](https://github.com/tomsing1) , [@tony-aw](https://github.com/tony-aw) , [@trevorld](https://github.com/trevorld) , [@tylerlittlefield](https://github.com/tylerlittlefield) , [@uriahf](https://github.com/uriahf) , [@urswilke](https://github.com/urswilke) , [@ValValetl](https://github.com/ValValetl) , [@venpopov](https://github.com/venpopov) , [@vincentvanhees](https://github.com/vincentvanhees) , [@wangq13](https://github.com/wangq13) , [@willgearty](https://github.com/willgearty) , [@wviechtb](https://github.com/wviechtb) , [@xuyiqing](https://github.com/xuyiqing) , [@yjunechoe](https://github.com/yjunechoe) , [@ynsec37](https://github.com/ynsec37) , [@zeehio](https://github.com/zeehio) , and [@zkamvar](https://github.com/zkamvar) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # pkgdown [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Pkgdown [![](/blog/2024/07/pkgdown-2-1-0/thumbnail-sq.jpg)](/blog/2024/07/pkgdown-2-1-0/) [pkgdown 2.1.0](/blog/2024/07/pkgdown-2-1-0/) [package](/categories/package) Hadley Wickham pkgdown 2.1.0 includes two major new features: support for quarto vignettes and a “light switch” that lets the reader switch between light and dark mode. It also contains a bunch of other improvements to both the user and the developer experience. [Read more ...](/blog/2024/07/pkgdown-2-1-0/) 2024/07/08 [![](/blog/2021/12/pkgdown-2-0-0/thumbnail-sq.jpg)](/blog/2021/12/pkgdown-2-0-0/) [pkgdown 2.0.0](/blog/2021/12/pkgdown-2-0-0/) [package](/categories/package) Hadley Wickham pkgdown 2.0.0 includes a major refresh of the default template (now using bootstrap 5), many new ways to customise your site, improvements to code styling, and much, much, more. [Read more ...](/blog/2021/12/pkgdown-2-0-0/) 2021/12/03 [![](/blog/2020/09/pkgdown-1-6-0/thumbnail-sq.jpg)](/blog/2020/09/pkgdown-1-6-0/) [pkgdown 1.6.0](/blog/2020/09/pkgdown-1-6-0/) [package](/categories/package) Hadley Wickham This release mostly contains bug fixes and minor improvements, but it now uses the downlit and ragg packages for syntax highlighting and graphical output, respectively. [Read more ...](/blog/2020/09/pkgdown-1-6-0/) 2020/09/12 [![](/blog/2020/03/pkgdown-1-5-0/thumbnail-sq.jpg)](/blog/2020/03/pkgdown-1-5-0/) [pkgdown 1.5.0](/blog/2020/03/pkgdown-1-5-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2020/03/pkgdown-1-5-0/) 2020/03/25 [![](/blog/2019/09/pkgdown-1-4-0/pkgdown-1-4-0-sq.jpg)](/blog/2019/09/pkgdown-1-4-0/) [pkgdown 1.4.0](/blog/2019/09/pkgdown-1-4-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2019/09/pkgdown-1-4-0/) 2019/09/05 [![](/blog/2018/12/pkgdown-1-3-0/pkgdown-1-3-0-sq.jpg)](/blog/2018/12/pkgdown-1-3-0/) [pkgdown 1.3.0](/blog/2018/12/pkgdown-1-3-0/) [package](/categories/package) Mara Averick pkgdown 1.3.0 is now on CRAN! [Read more ...](/blog/2018/12/pkgdown-1-3-0/) 2018/12/10 [![](/blog/2018/06/pkgdown-1-1-0/pkgdown-1-1-0-sq.jpg)](/blog/2018/06/pkgdown-1-1-0/) [pkgdown 1.1.0](/blog/2018/06/pkgdown-1-1-0/) [package](/categories/package) Mara Averick pkgdown 1.1.0 is now on CRAN. [Read more ...](/blog/2018/06/pkgdown-1-1-0/) 2018/06/05 [![](/blog/2018/05/pkgdown-1-0-0/pkgdown-1-0-0-sq.jpg)](/blog/2018/05/pkgdown-1-0-0/) [pkgdown 1.0.0](/blog/2018/05/pkgdown-1-0-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2018/05/pkgdown-1-0-0/) 2018/05/03 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # recipes 1.1.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) recipes 1.1.0 ============= ![](/blog/2024/07/recipes-1-1-0/thumbnail-wd.jpg) Photo by [Food Photographer | Jennifer Pallian](https://unsplash.com/photos/close-up-photo-of-baked-cookies-OfdDiqx8Cz8)   2024/07/08   [tidymodels](/tags/tidymodels/) , [recipes](/tags/recipes/)   Emil Hvitfeldt We’re thrilled to announce the release of [recipes](https://recipes.tidymodels.org/) 1.1.0. recipes lets you create a pipeable sequence of feature engineering steps. You can install it from CRAN with: install.packages("recipes") This blog post will go over some of the bigger changes in this release. Improvements in column type checking, allowing more data types to be passed to recipes, use of long formulas and better error for misspelled argument names. You can see a full list of changes in the [release notes](https://github.com/tidymodels/recipes/releases/tag/v1.1.0) . Column type checking[](#column-type-checking) ---------------------------------------------- A [longtime issue](https://github.com/tidymodels/recipes/issues/793) in recipes came from the fact that recipes didn’t keep a [prototype](https://vctrs.r-lib.org/articles/type-size.html) (ptype) of the data it was specified with. This would cause unexpected things to happen or uninformative error messages to appear if different data was used to [`prep()`](https://recipes.tidymodels.org/reference/prep.html) than was used to create the [`recipe()`](https://recipes.tidymodels.org/reference/recipe.html) . Every recipe you create starts with a call to [`recipe()`](https://recipes.tidymodels.org/reference/recipe.html) . In the below example, we create a recipe where `x2` starts by being a character vector, but the recipe is prepped where `x2` is a numeric vector. This didn’t produce any warnings or errors, silently doing something unintended. data_template <- tibble( outcome = rnorm(10), x1 = rnorm(10), x2 = sample(letters, 10, T) ) rec <- recipe(outcome ~ ., data_template) %>% step_bin2factor(all_numeric_predictors()) data_training <- tibble(outcome = rnorm(1000), x1 = rnorm(1000), x2 = rnorm(1000)) prep(rec, training = data_training) #> #> ── Recipe ────────────────────────────────────────────────────────────────────── #> #> ── Inputs #> Number of variables by role #> outcome: 1 #> predictor: 2 #> #> ── Training information #> Training data contained 1000 data points and no incomplete rows. #> #> ── Operations #> • Dummy variable to factor conversion for: x1 | Trained Now, we get an error detailing how the data is different. data_template <- tibble(outcome = rnorm(10), x1 = rnorm(10), x2 = sample(letters, 10, T)) rec <- recipe(outcome ~ ., data_template) %>% step_bin2factor(all_numeric_predictors()) data_training <- tibble(outcome = rnorm(1000), x1 = rnorm(1000), x2 = rnorm(1000)) prep(rec, training = data_training) #> Error in `prep()`: #> ✖ The following variable has the wrong class: #> • `x2` must have class , not . Note that recipes created before version 1.1.0 don’t contain any ptype information, and will not undergo checking. Rerunning the code to create the recipe will add ptype information to the recipe. Input checking in `recipe()`[](#input-checking-in-recipe) ---------------------------------------------------------- We have relaxed the requirements of data frames, while making feedback more helpful when something goes wrong. The data was previously passed through [`model.frame()`](https://rdrr.io/r/stats/model.frame.html) inside the recipe, which restricted what could be handled. Previously prohibited input included data frames with list-columns or [sf](https://r-spatial.github.io/sf/) data frames. Both of these are now supported, as long as they are a `data.frame` object. data_listcolumn <- tibble( y = 1:4, x = list(1:3, 4:6, 3:1, 1:10) ) recipe(y ~ ., data = data_listcolumn) #> #> ── Recipe ────────────────────────────────────────────────────────────────────── #> #> ── Inputs #> Number of variables by role #> outcome: 1 #> predictor: 1 library(sf) #> Linking to GEOS 3.11.0, GDAL 3.5.3, PROJ 9.1.0; sf_use_s2() is TRUE pathshp <- system.file("shape/nc.shp", package = "sf") data_sf <- st_read(pathshp, quiet = TRUE) recipe(AREA ~ ., data = data_sf) #> #> ── Recipe ────────────────────────────────────────────────────────────────────── #> #> ── Inputs #> Number of variables by role #> outcome: 1 #> predictor: 14 We are excited to see what people can do with these new options. Another way to tell a recipe what variables should be included and what roles they should have is to use [`add_role()`](https://recipes.tidymodels.org/reference/roles.html) and [`update_role()`](https://recipes.tidymodels.org/reference/roles.html) . But if you were not careful, you could end up in situations where the same variable is labeled as both the outcome and predictor. # didn't used to throw a warning recipe(mtcars) |> update_role(everything(), new_role = "predictor") |> add_role("mpg", new_role = "outcome") #> Error in `add_role()`: #> ! `mpg` cannot get "outcome" role as it already has role "predictor". This error can be avoided by using [`update_role()`](https://recipes.tidymodels.org/reference/roles.html) instead of [`add_role()`](https://recipes.tidymodels.org/reference/roles.html) . recipe(mtcars) |> update_role(everything(), new_role = "predictor") |> update_role("mpg", new_role = "outcome") #> #> ── Recipe ────────────────────────────────────────────────────────────────────── #> #> ── Inputs #> Number of variables by role #> outcome: 1 #> predictor: 10 Long formulas in `recipe()`[](#long-formulas-in-recipe) -------------------------------------------------------- Related to the changes we saw above, we now fully support very long formulas without hitting a `C stack usage` error. data_wide <- matrix(1:10000, ncol = 10000) data_wide <- as.data.frame(data_wide) names(data_wide) <- c(paste0("x", 1:10000)) long_formula <- as.formula(paste("~ ", paste(names(data_wide), collapse = " + "))) recipe(long_formula, data_wide) #> #> ── Recipe ────────────────────────────────────────────────────────────────────── #> #> ── Inputs #> Number of variables by role #> predictor: 10000 Better error for misspelled argument names[](#better-error-for-misspelled-argument-names) ------------------------------------------------------------------------------------------ If you have used recipes long enough you are very likely to have run into the following error. recipe(mpg ~ ., data = mtcars) |> step_pca(all_numeric_predictors(), number = 4) |> prep() #> Error in `step_pca()`: #> Caused by error in `prep()`: #> ! Can't rename variables in this context. The first time you saw it, it didn’t make much sense. Hopefully, you figured out that [step\_pca()](https://recipes.tidymodels.org/reference/step_pca.html) doesn’t have a `number` argument, and instead uses `num_comp` to determine the number of principal components to return. This confusion will be a thing of the past as we now include this improved error message. recipe(mpg ~ ., data = mtcars) |> step_pca(all_numeric_predictors(), number = 4) |> prep() #> Error in `step_pca()`: #> Caused by error in `prep()` at recipes/R/recipe.R:479:9: #> ! The following argument was specified but do not exist: `number`. Quality of life increases in `step_dummy()`[](#quality-of-life-increases-in-step_dummy) ---------------------------------------------------------------------------------------- I would imagine that one of the most used steps is [`step_dummy()`](https://recipes.tidymodels.org/reference/step_dummy.html) . We have improved the errors and warnings it spits out when things go sideways. If you apply [`step_dummy()`](https://recipes.tidymodels.org/reference/step_dummy.html) to a variable that contains a lot of levels, it will produce a lot of columns, and the resulting object may not fit in memory. This can lead to the following error. data_id <- tibble( id = as.character(1:100000), x1 = rnorm(100000), x2 = sample(letters, 100000, TRUE) ) recipe(~ ., data = data_id) |> step_dummy(all_nominal_predictors()) |> prep() #> Error: vector memory exhausted (limit reached?) Instead, you now get a more helpful error message. data_id <- tibble( id = as.character(1:100000), x1 = rnorm(100000), x2 = sample(letters, 100000, TRUE) ) recipe(~ ., data = data_id) |> step_dummy(all_nominal_predictors()) |> prep() #> Error in `step_dummy()`: #> Caused by error: #> ! `id` contains too many levels (100000), which would result in a #> data.frame too large to fit in memory. Likewise, you will get helpful errors if [`step_dummy()`](https://recipes.tidymodels.org/reference/step_dummy.html) gets a `NA` or unseen values. data_train <- tibble(x = c("a", "b")) data_unseen <- tibble(x = "c") rec_spec <- recipe(~., data = data_train) %>% step_dummy(x) %>% prep() rec_spec %>% bake(data_unseen) #> Warning: ! There are new levels in `x`: "c". #> ℹ Consider using step_novel() (`?recipes::step_novel()`) before `step_dummy()` #> to handle unseen values. #> # A tibble: 1 × 1 #> x_b #> #> 1 NA data_na <- tibble(x = NA) rec_spec %>% bake(data_na) #> Warning: ! There are new levels in `x`: NA. #> ℹ Consider using step_unknown() (`?recipes::step_unknown()`) before #> `step_dummy()` to handle missing values. #> # A tibble: 1 × 1 #> x_b #> #> 1 NA Acknowledgements[](#acknowledgements) -------------------------------------- A big thank you to all the people who have contributed to recipes since the release of v1.0.10: [@brynhum](https://github.com/brynhum) , [@DemetriPananos](https://github.com/DemetriPananos) , [@diegoperoni](https://github.com/diegoperoni) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@JiahuaQu](https://github.com/JiahuaQu) , [@joranE](https://github.com/joranE) , [@nhward](https://github.com/nhward) , [@olivroy](https://github.com/olivroy) , and [@simonpcouch](https://github.com/simonpcouch) . Chocolate Chocolate Chip Cookies[](#chocolate-chocolate-chip-cookies) ---------------------------------------------------------------------- preheat oven 350°F * 1/3c butter * 1/2 + 1/3c sugar mix until fluffy * 1 tsp vanilla * 1 egg mix until combined * 1/2c cocoa * 1/2 tsp baking soda * 1c flour mix until combined * 3/4c chocolate chips bake for about 8 mins, depending on size! they will crack on top, but still be soft. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # bonsai [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Bonsai [![](/blog/2024/06/bonsai-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/bonsai-0-3-0/) [bonsai 0.3.0](/blog/2024/06/bonsai-0-3-0/) [package](/categories/package) Simon Couch A new release of the parsnip extension package bonsai introduces support for oblique random forests for classification and regression to tidymodels. [Read more ...](/blog/2024/06/bonsai-0-3-0/) 2024/06/25 [![](/blog/2022/06/bonsai-0-1-0/thumbnail-sq.jpg)](/blog/2022/06/bonsai-0-1-0/) [bonsai 0.1.0](/blog/2022/06/bonsai-0-1-0/) [package](/categories/package) Simon Couch A new parsnip extension package for tree-based models is now on CRAN. [Read more ...](/blog/2022/06/bonsai-0-1-0/) 2022/06/30 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # parsnip [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Parsnip [![](/blog/2024/06/bonsai-0-3-0/thumbnail-sq.jpg)](/blog/2024/06/bonsai-0-3-0/) [bonsai 0.3.0](/blog/2024/06/bonsai-0-3-0/) [package](/categories/package) Simon Couch A new release of the parsnip extension package bonsai introduces support for oblique random forests for classification and regression to tidymodels. [Read more ...](/blog/2024/06/bonsai-0-3-0/) 2024/06/25 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2023/11/tidymodels-errors-q4/thumbnail-sq.jpg)](/blog/2023/11/tidymodels-errors-q4/) [Three ways errors are about to get better in tidymodels](/blog/2023/11/tidymodels-errors-q4/) [programming](/categories/programming) Simon Couch The tidymodels team’s biannual spring cleaning gave us a chance to revisit the way we raise some error messages. [Read more ...](/blog/2023/11/tidymodels-errors-q4/) 2023/11/10 [![](/blog/2023/04/censored-0-2-0/thumbnail-sq.jpg)](/blog/2023/04/censored-0-2-0/) [censored 0.2.0](/blog/2023/04/censored-0-2-0/) [package](/categories/package) Hannah Frick censored 0.2.0 is on CRAN! censored has two new engines for random forests and parametric survival models. [Read more ...](/blog/2023/04/censored-0-2-0/) 2023/04/19 [![](/blog/2022/12/tidymodels-2022-q4/thumbnail-sq.jpg)](/blog/2022/12/tidymodels-2022-q4/) [Q4 2022 tidymodels digest](/blog/2022/12/tidymodels-2022-q4/) [roundup](/categories/roundup) Simon Couch The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2022/12/tidymodels-2022-q4/) 2022/12/29 [![](/blog/2022/10/parsnip-checking-1-0-2/thumbnail-sq.jpg)](/blog/2022/10/parsnip-checking-1-0-2/) [Improvements to model specification checking in tidymodels](/blog/2022/10/parsnip-checking-1-0-2/) [package](/categories/package) Simon Couch parsnip 1.0.2 includes a number of changes to how the package checks model specifications, improving error messages and tightening integration with its extension packages. [Read more ...](/blog/2022/10/parsnip-checking-1-0-2/) 2022/10/03 [![](/blog/2022/08/censored-0-1-0/thumbnail-sq.jpg)](/blog/2022/08/censored-0-1-0/) [censored 0.1.0](/blog/2022/08/censored-0-1-0/) [package](/categories/package) Hannah Frick censored 0.1.0 is a new tidymodels package for survival models. [Read more ...](/blog/2022/08/censored-0-1-0/) 2022/08/10 [![](/blog/2022/07/tidymodels-2022-q2/thumbnail-sq.jpg)](/blog/2022/07/tidymodels-2022-q2/) [Q2 2022 tidymodels digest](/blog/2022/07/tidymodels-2022-q2/) [roundup](/categories/roundup) Emil Hvitfeldt Q2 marks the end of the season of case weights, with 25 new releases. [Read more ...](/blog/2022/07/tidymodels-2022-q2/) 2022/07/19 [![](/blog/2022/06/bonsai-0-1-0/thumbnail-sq.jpg)](/blog/2022/06/bonsai-0-1-0/) [bonsai 0.1.0](/blog/2022/06/bonsai-0-1-0/) [package](/categories/package) Simon Couch A new parsnip extension package for tree-based models is now on CRAN. [Read more ...](/blog/2022/06/bonsai-0-1-0/) 2022/06/30 [![](/blog/2022/05/case-weights/thumbnail-sq.jpg)](/blog/2022/05/case-weights/) [Using case weights with tidymodels](/blog/2022/05/case-weights/) [deep-dive](/categories/deep-dive) Max Kuhn Support for case weights is now available across many tidymodels packages. [Read more ...](/blog/2022/05/case-weights/) 2022/05/05 * [««](/tags/parsnip/) * « * [1](/tags/parsnip/) * [2](/tags/parsnip/page/2/) * [3](/tags/parsnip/page/3/) * [»](/tags/parsnip/page/2/) * [»»](/tags/parsnip/page/3/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # parallelism [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Parallelism [![](/blog/2024/04/tune-1-2-0/thumbnail-sq.jpg)](/blog/2024/04/tune-1-2-0/) [tune 1.2.0](/blog/2024/04/tune-1-2-0/) [package](/categories/package) Simon Couch While we’ve written about survival analysis and machine learning fairness already, the newest tune release includes a number of other major changes. [Read more ...](/blog/2024/04/tune-1-2-0/) 2024/04/18 [![](/blog/2020/11/tune-parallel/thumbnail-sq.jpg)](/blog/2020/11/tune-parallel/) [Parallel processing with tune](/blog/2020/11/tune-parallel/) [package](/categories/package) Max Kuhn With version 0.1.2 of tune, there are more options for parallel processing. [Read more ...](/blog/2020/11/tune-parallel/) 2020/11/27 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # bonsai 0.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) bonsai 0.3.0 ============ ![](/blog/2024/06/bonsai-0-3-0/thumbnail-wd.jpg) Photo by [Scott Webb](https://unsplash.com/photos/QD-mtViB5Ks)   2024/06/25   [tidymodels](/tags/tidymodels/) , [bonsai](/tags/bonsai/) , [parsnip](/tags/parsnip/)   Simon Couch We’re brimming with glee to announce the release of [bonsai](https://bonsai.tidymodels.org) 0.3.0. bonsai is a parsnip extension package for tree-based models, and includes support for random forest and gradient-boosted tree frameworks like partykit and LightGBM. This most recent release of the package introduces support for the `"aorsf"` engine, which implements accelerated oblique random forests (Jaeger et al. 2022, Jaeger et al. 2024). You can install it from CRAN with: install.packages("bonsai") This blog post will demonstrate a modeling workflow where the benefits of using oblique random forests shine through. You can see a full list of changes in the [release notes](https://bonsai.tidymodels.org/news/index.html#bonsai-030) . library(tidymodels) library(bonsai) library(plsmod) library(corrr) The `meats` data[](#the-meats-data) ------------------------------------ The modeldata package, loaded automatically with the tidymodels meta-package, includes several example datasets to demonstrate modeling problems. We’ll make use of a dataset called `meats` in this post. Each row is a measurement of a sample of finely chopped meat. meats #> # A tibble: 215 × 103 #> x_001 x_002 x_003 x_004 x_005 x_006 x_007 x_008 x_009 x_010 x_011 x_012 x_013 #> #> 1 2.62 2.62 2.62 2.62 2.62 2.62 2.62 2.62 2.63 2.63 2.63 2.63 2.64 #> 2 2.83 2.84 2.84 2.85 2.85 2.86 2.86 2.87 2.87 2.88 2.88 2.89 2.90 #> 3 2.58 2.58 2.59 2.59 2.59 2.59 2.59 2.60 2.60 2.60 2.60 2.61 2.61 #> 4 2.82 2.82 2.83 2.83 2.83 2.83 2.83 2.84 2.84 2.84 2.84 2.85 2.85 #> 5 2.79 2.79 2.79 2.79 2.80 2.80 2.80 2.80 2.81 2.81 2.81 2.82 2.82 #> 6 3.01 3.02 3.02 3.03 3.03 3.04 3.04 3.05 3.06 3.06 3.07 3.08 3.09 #> 7 2.99 2.99 3.00 3.01 3.01 3.02 3.02 3.03 3.04 3.04 3.05 3.06 3.07 #> 8 2.53 2.53 2.53 2.53 2.53 2.53 2.53 2.53 2.54 2.54 2.54 2.54 2.54 #> 9 3.27 3.28 3.29 3.29 3.30 3.31 3.31 3.32 3.33 3.33 3.34 3.35 3.36 #> 10 3.40 3.41 3.41 3.42 3.43 3.43 3.44 3.45 3.46 3.47 3.48 3.48 3.49 #> # ℹ 205 more rows #> # ℹ 90 more variables: x_014 , x_015 , x_016 , x_017 , #> # x_018 , x_019 , x_020 , x_021 , x_022 , #> # x_023 , x_024 , x_025 , x_026 , x_027 , #> # x_028 , x_029 , x_030 , x_031 , x_032 , #> # x_033 , x_034 , x_035 , x_036 , x_037 , #> # x_038 , x_039 , x_040 , x_041 , x_042 , … From that dataset’s documentation: > These data are recorded on a Tecator Infratec Food and Feed Analyzer… For each meat sample the data consists of a 100 channel spectrum of absorbances and the contents of moisture (water), fat and protein. The absorbance is -log10 of the transmittance measured by the spectrometer. The three contents, measured in percent, are determined by analytic chemistry. We’ll try to predict the protein content, as a percentage, using the absorbance measurements. Before we take a further look, let’s split up our data. I’ll first select off two other possible outcome variables and, after splitting into training and testing sets, resample the data using 5-fold cross-validation with 2 repeats. meats <- meats %>% select(-water, -fat) set.seed(1) meats_split <- initial_split(meats) meats_train <- training(meats_split) meats_test <- testing(meats_split) meats_folds <- vfold_cv(meats_train, v = 5, repeats = 2) The tricky parts of this modeling problem are that: 1. There are few observations to work with (215 total). 2. Each of these 100 absorbance measurements are _highly_ correlated. Visualizing that correlation: meats_train %>% correlate() %>% autoplot() + theme(axis.text.x = element_blank(), axis.text.y = element_blank()) #> Correlation computed with #> • Method: 'pearson' #> • Missing treated using: 'pairwise.complete.obs' ![](figs/correlate-1.png) Almost all of these pairwise correlations between predictors are near 1, besides the last variable and every other variable. That last variable with weaker correlation values? It’s the outcome. Baseline models[](#baseline-models) ------------------------------------ There are several existing model implementations in tidymodels that are resilient to highly correlated predictors. The first one I’d probably reach for is an elastic net: an interpolation of the LASSO and Ridge regularized linear regression models. Evaluating that modeling approach against resamples: # define a regularized linear model spec_lr <- linear_reg(penalty = tune(), mixture = tune()) %>% set_engine("glmnet") # try out different penalization approaches res_lr <- tune_grid(spec_lr, protein ~ ., meats_folds) show_best(res_lr, metric = "rmse") #> # A tibble: 5 × 8 #> penalty mixture .metric .estimator mean n std_err .config #> #> 1 0.0000324 0.668 rmse standard 1.24 10 0.0516 Preprocessor1_Mo… #> 2 0.00000000524 0.440 rmse standard 1.25 10 0.0548 Preprocessor1_Mo… #> 3 0.000000461 0.839 rmse standard 1.26 10 0.0538 Preprocessor1_Mo… #> 4 0.00000550 0.965 rmse standard 1.26 10 0.0540 Preprocessor1_Mo… #> 5 0.0000000489 0.281 rmse standard 1.26 10 0.0534 Preprocessor1_Mo… show_best(res_lr, metric = "rsq") #> # A tibble: 5 × 8 #> penalty mixture .metric .estimator mean n std_err .config #> #> 1 0.0000324 0.668 rsq standard 0.849 10 0.0126 Preprocessor1_Mo… #> 2 0.00000000524 0.440 rsq standard 0.848 10 0.0128 Preprocessor1_Mo… #> 3 0.000000461 0.839 rsq standard 0.846 10 0.0114 Preprocessor1_Mo… #> 4 0.00000550 0.965 rsq standard 0.846 10 0.0111 Preprocessor1_Mo… #> 5 0.0000000489 0.281 rsq standard 0.846 10 0.0126 Preprocessor1_Mo… That best RMSE value of 1.24 gives us a baseline to work with, and the best R-squared 0.85 seems like a good start. Many tree-based model implementations in tidymodels generally handle correlated predictors well. Just to be apples-to-apples with `"aorsf"`, let’s use a different random forest engine to get a better sense for baseline performance: spec_rf <- rand_forest(mtry = tune(), min_n = tune()) %>% # this is the default engine, but for consistency's sake: set_engine("ranger") %>% set_mode("regression") res_rf <- tune_grid(spec_rf, protein ~ ., meats_folds) #> i Creating pre-processing data to finalize unknown parameter: mtry show_best(res_rf, metric = "rmse") #> # A tibble: 5 × 8 #> mtry min_n .metric .estimator mean n std_err .config #> #> 1 96 4 rmse standard 2.37 10 0.0905 Preprocessor1_Model08 #> 2 41 6 rmse standard 2.39 10 0.0883 Preprocessor1_Model01 #> 3 88 10 rmse standard 2.43 10 0.0816 Preprocessor1_Model06 #> 4 79 17 rmse standard 2.51 10 0.0740 Preprocessor1_Model07 #> 5 27 18 rmse standard 2.52 10 0.0778 Preprocessor1_Model04 show_best(res_rf, metric = "rsq") #> # A tibble: 5 × 8 #> mtry min_n .metric .estimator mean n std_err .config #> #> 1 96 4 rsq standard 0.424 10 0.0385 Preprocessor1_Model08 #> 2 41 6 rsq standard 0.409 10 0.0394 Preprocessor1_Model01 #> 3 88 10 rsq standard 0.387 10 0.0365 Preprocessor1_Model06 #> 4 79 17 rsq standard 0.353 10 0.0404 Preprocessor1_Model07 #> 5 27 18 rsq standard 0.346 10 0.0397 Preprocessor1_Model04 Not so hot. Just to show I’m not making a straw man here, I’ll evaluate a few more alternative modeling approaches behind the curtain and print out their best performance metrics: * **Gradient boosted tree with LightGBM**. Best RMSE: 2.34. Best R-squared: 0.43. * **Partial least squares regression**. Best RMSE: 1.39. Best R-squared: 0.81. * **Support vector machine**. Best RMSE: 2.28. Best R-squared: 0.46. This is a tricky one. Introducing accelerated oblique random forests[](#introducing-accelerated-oblique-random-forests) -------------------------------------------------------------------------------------------------- The 0.3.0 release of bonsai introduces support for accelerated oblique random forests via the `"aorsf"` engine for classification and regression in tidymodels. (Tidy survival modelers might note that [we already support `"aorsf"` for censored regression](https://www.tidyverse.org/blog/2023/04/censored-0-2-0/) via the [censored](https://censored.tidymodels.org) parsnip extension package!) Unlike trees in conventional random forests, which create splits using thresholds based on individual predictors (e.g. `x_001 > 3`), oblique random forests use linear combinations of predictors to create splits (e.g. `x_001 * x_002 > 7.5`) and have been shown to improve predictive performance related to conventional random forests for a variety of applications (Menze et al. 2011). “Oblique” references the appearance of decision boundaries when a set of splits is plotted; I’ve grabbed a visual from the [aorsf README](https://github.com/ropensci/aorsf?tab=readme-ov-file#what-does-oblique-mean) that demonstrates: ![Two plots of decision boundaries for a classification problem. One uses single-variable splitting and the other oblique splitting. Both trees partition the predictor space defined by predictors X1 and X2, but the oblique splits do a better job of separating the two classes thanks to an 'oblique' boundary formed by considering both X1 and X2 at the same time.](figures/oblique.png) In the above, we’d like to separate the purple dots from the orange squares. A tree in a traditional random forest, represented on the left, can only generate splits based on one of X1 or X2 at a time. A tree in an oblique random forest, represented on the right, can consider both X1 and X2 in creating decision boundaries, often resulting in stronger predictive performance. Where does the “accelerated” come from? Generally, finding optimal oblique splits is computationally more intensive than finding single-predictor splits. The aorsf package uses something called “Newton Raphson scoring”—the same algorithm under the hood in the survival package—to identify splits based on linear combinations of predictor variables. This approach speeds up that process greatly, resulting in fit times that are analogous to implementations of traditional random forests in R (and hundreds of times faster than existing oblique random forest implementations, Jaeger et al. 2024). The code to tune this model with the `"aorsf"` engine is the same as for `"ranger"`, except we switch out the `engine` argument to [`set_engine()`](https://parsnip.tidymodels.org/reference/set_engine.html) : spec_aorsf <- rand_forest( mtry = tune(), min_n = tune() ) %>% set_engine("aorsf") %>% set_mode("regression") res_aorsf <- tune_grid(spec_aorsf, protein ~ ., meats_folds) #> i Creating pre-processing data to finalize unknown parameter: mtry show_best(res_aorsf, metric = "rmse") #> # A tibble: 5 × 8 #> mtry min_n .metric .estimator mean n std_err .config #> #> 1 87 11 rmse standard 0.786 10 0.0370 Preprocessor1_Model02 #> 2 98 8 rmse standard 0.789 10 0.0363 Preprocessor1_Model10 #> 3 48 5 rmse standard 0.793 10 0.0363 Preprocessor1_Model01 #> 4 16 17 rmse standard 0.803 10 0.0325 Preprocessor1_Model09 #> 5 31 18 rmse standard 0.813 10 0.0359 Preprocessor1_Model05 show_best(res_aorsf, metric = "rsq") #> # A tibble: 5 × 8 #> mtry min_n .metric .estimator mean n std_err .config #> #> 1 48 5 rsq standard 0.946 10 0.00446 Preprocessor1_Model01 #> 2 98 8 rsq standard 0.945 10 0.00482 Preprocessor1_Model10 #> 3 87 11 rsq standard 0.945 10 0.00484 Preprocessor1_Model02 #> 4 16 17 rsq standard 0.941 10 0.00370 Preprocessor1_Model09 #> 5 31 18 rsq standard 0.940 10 0.00547 Preprocessor1_Model05 Holy smokes. The best RMSE from aorsf is 0.79, much more performant than the previous best RMSE from the elastic net with a value of 1.24, and the best R-squared is 0.95, much stronger than the previous best (also from the elastic net) of 0.85. Especially if your modeling problems involve few samples of many, highly correlated predictors, give the `"aorsf"` modeling engine a whirl in your workflows and let us know what you think! References[](#references) -------------------------- Byron C. Jaeger, Sawyer Welden, Kristin Lenoir, Jaime L. Speiser, Matthew W. Segar, Ambarish Pandey, Nicholas M. Pajewski. 2024. “Accelerated and Interpretable Oblique Random Survival Forests.” _Journal of Computational and Graphical Statistics_ 33.1: 192-207. Byron C. Jaeger, Sawyer Welden, Kristin Lenoir, and Nicholas M. Pajewski. 2022. “aorsf: An R package for Supervised Learning Using the Oblique Random Survival Forest.” _The Journal of Open Source Software_. Bjoern H. Menze, B. Michael Kelm, Daniel N. Splitthoff, Ullrich Koethe, and Fred A. Hamprecht. (2011). “On Oblique Random Forests.” _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ (pp. 453–469). Springer. Acknowledgements[](#acknowledgements) -------------------------------------- Thank you to [@bcjaeger](https://github.com/bcjaeger) , the aorsf author, for doing most of the work to implement aorsf support in bonsai. Thank you to [@hfrick](https://github.com/hfrick) , [@joranE](https://github.com/joranE) , [@jrosell](https://github.com/jrosell) , [@nipnipj](https://github.com/nipnipj) , [@p-schaefer](https://github.com/p-schaefer) , [@seb-mueller](https://github.com/seb-mueller) , and [@tcovert](https://github.com/tcovert) for their contributions on the bonsai repository since version 0.2.1. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # censored [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Censored [![](/blog/2024/04/tidymodels-2024-q1/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-2024-q1/) [Q1 2024 tidymodels digest](/blog/2024/04/tidymodels-2024-q1/) [roundup](/categories/roundup) Hannah Frick The tidymodels team has been busy working on all sorts of new features across the framework. [Read more ...](/blog/2024/04/tidymodels-2024-q1/) 2024/04/24 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2023/04/censored-0-2-0/thumbnail-sq.jpg)](/blog/2023/04/censored-0-2-0/) [censored 0.2.0](/blog/2023/04/censored-0-2-0/) [package](/categories/package) Hannah Frick censored 0.2.0 is on CRAN! censored has two new engines for random forests and parametric survival models. [Read more ...](/blog/2023/04/censored-0-2-0/) 2023/04/19 [![](/blog/2022/08/censored-0-1-0/thumbnail-sq.jpg)](/blog/2022/08/censored-0-1-0/) [censored 0.1.0](/blog/2022/08/censored-0-1-0/) [package](/categories/package) Hannah Frick censored 0.1.0 is a new tidymodels package for survival models. [Read more ...](/blog/2022/08/censored-0-1-0/) 2022/08/10 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # workflowsets [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Workflowsets [![](/blog/2024/04/tidymodels-2024-q1/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-2024-q1/) [Q1 2024 tidymodels digest](/blog/2024/04/tidymodels-2024-q1/) [roundup](/categories/roundup) Hannah Frick The tidymodels team has been busy working on all sorts of new features across the framework. [Read more ...](/blog/2024/04/tidymodels-2024-q1/) 2024/04/24 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2023/04/tuning-delights/thumbnail-sq.jpg)](/blog/2023/04/tuning-delights/) [Tuning hyperparameters with tidymodels is a delight](/blog/2023/04/tuning-delights/) [roundup](/categories/roundup) Simon Couch New releases of the tune, finetune, and workflowsets packages have made optimizing model parameters with tidymodels even more pleasant. [Read more ...](/blog/2023/04/tuning-delights/) 2023/04/20 [![](/blog/2022/04/tidymodels-2022-q1/thumbnail-sq.jpg)](/blog/2022/04/tidymodels-2022-q1/) [Q1 2022 tidymodels digest](/blog/2022/04/tidymodels-2022-q1/) [roundup](/categories/roundup) Julia Silge There were 21 releases of tidymodels packages during Q1 of this year, and this post shares some highlights! [Read more ...](/blog/2022/04/tidymodels-2022-q1/) 2022/04/01 [![](/blog/2021/07/tidymodels-july-2021/thumbnail-sq.jpg)](/blog/2021/07/tidymodels-july-2021/) [New tidymodels releases for July 2021](/blog/2021/07/tidymodels-july-2021/) [roundup](/categories/roundup) Max Kuhn We highlight a series of new tidymodels package versions and their improvements. [Read more ...](/blog/2021/07/tidymodels-july-2021/) 2021/07/27 [![](/blog/2021/05/choose-tidymodels-adventure/thumbnail-sq.jpg)](/blog/2021/05/choose-tidymodels-adventure/) [Choose your own tidymodels adventure](/blog/2021/05/choose-tidymodels-adventure/) [learn](/categories/learn) Julia Silge The tidymodels ecosystem is modular and flexible, which can sometimes make choosing an approach overwhelming for newcomers. This post offers opinionated guidance on where to start! [Read more ...](/blog/2021/05/choose-tidymodels-adventure/) 2021/05/24 [![](/blog/2021/03/workflowsets-0-0-1/thumbnail-sq.jpg)](/blog/2021/03/workflowsets-0-0-1/) [workflowsets 0.0.1](/blog/2021/03/workflowsets-0-0-1/) [package](/categories/package) Max Kuhn A new package for evaluating many models at once is now on CRAN, along with a corresponding update to the tidyposterior package. [Read more ...](/blog/2021/03/workflowsets-0-0-1/) 2021/03/30 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Tidyverse developer day 2024 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyverse developer day 2024 ============================ ![](/blog/2024/04/tdd-2024/thumbnail-wd.jpg) Photo by [JD Long](https://cerebralmastication.com)   2024/04/09   [tidyverse-dev-day](/tags/tidyverse-dev-day/)   Hadley Wickham It’s been a hot minute since the last one, but we are very excited to announce that the next tidyverse developer day will be after [posit::conf](https://posit.co/conference/) in Seattle on August 15, 2024. A big thanks goes to [Fred Hutch Cancer Center](https://www.fredhutch.org/en.html) for donating the space! **What is the tidyverse developer day?** TDD is a day of learning and coding to nurture regular contributors to the tidyverse. We’ll provide food; you’ll bring your laptop and enthusiasm. The tidyverse team and other community helpers will be on hand to help you hit the ground running and/or get over any stumbling blocks that you encounter. Don’t have any ideas for something to work on? No problem! We’ll be tagging issues in advance to make sure there’s lots to do for any- and everyone, regardless of level of expertise. **Who should attend?** Anyone who would like to get better at contributing to the tidyverse! Everyone is welcome regardless of whether you’ve never done a PR before, or if you’ve already made your 10th package. But you do need a ticket; to provide a fulfilling experience for all attendees we need to carefully manage the ratio of attendees to helpers. **How much does it cost?** $10. This doesn’t cover the costs of the event because we don’t want to make attendance contingent on your ability to pay, but we’ve found some monetary commitment discourages people from taking tickets that they don’t end up using. But if the cost would prevent you from attending, please email [jenny@posit.co](mailto:jenny@posit.co) and we can figure something out. [Buy your ticket now!](https://www.eventbrite.com/e/tidyverse-developer-day-2024-tickets-876018203027?aff=oddtdtcreator) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dbplyr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Dbplyr [![](/blog/2024/04/dbplyr-2-5-0/thumbnail-sq.jpg)](/blog/2024/04/dbplyr-2-5-0/) [dbplyr 2.5.0](/blog/2024/04/dbplyr-2-5-0/) [package](/categories/package) Hadley Wickham dbplyr 2.5.0 brings improved syntax for referring to tables nested in schemas and catalogs along with a bunch of minor SQL generation improvements. [Read more ...](/blog/2024/04/dbplyr-2-5-0/) 2024/04/08 [![](/blog/2023/10/dbplyr-2-4-0/thumbnail-sq.jpg)](/blog/2023/10/dbplyr-2-4-0/) [dbplyr 2.4.0](/blog/2023/10/dbplyr-2-4-0/) [package](/categories/package) Hadley Wickham dbplyr 2.4.0 brings improvements to SQL generation, better control over the generated SQL, some new translations, and a bunch of backend specific improvements. [Read more ...](/blog/2023/10/dbplyr-2-4-0/) 2023/10/26 [![](/blog/2023/01/dbplyr-2-3-0/thumbnail-sq.jpg)](/blog/2023/01/dbplyr-2-3-0/) [dbplyr 2.3.0](/blog/2023/01/dbplyr-2-3-0/) [package](/categories/package) Hadley Wickham dbplyr 2.3.0 brings improvements to SQL generation, improved error messages, a handful of new translations, and a bunch of backend specific improvements. [Read more ...](/blog/2023/01/dbplyr-2-3-0/) 2023/01/16 [![](/blog/2022/06/dbplyr-2-2-0/thumbnail-sq.jpg)](/blog/2022/06/dbplyr-2-2-0/) [dbplyr 2.2.0](/blog/2022/06/dbplyr-2-2-0/) [package](/categories/package) Hadley Wickham This release brings improvements to SQL translation, a new way of getting local data into the database, and support for dplyr’s family of row modification functions. [Read more ...](/blog/2022/06/dbplyr-2-2-0/) 2022/06/06 [![](/blog/2021/08/bigrquery-1-4-0/thumbnail-sq.jpg)](/blog/2021/08/bigrquery-1-4-0/) [bigrquery 1.4.0](/blog/2021/08/bigrquery-1-4-0/) [package](/categories/package) Jenny Bryan bigrquery 1.4.0 fixes a bug in `bq_table_download()`. [Read more ...](/blog/2021/08/bigrquery-1-4-0/) 2021/08/04 [![](/blog/2020/11/dbplyr-2-0-0/thumbnail-sq.jpg)](/blog/2020/11/dbplyr-2-0-0/) [dbplyr 2.0.0](/blog/2020/11/dbplyr-2-0-0/) [package](/categories/package) Hadley Wickham dbplyr 2.0.0 adds missing features from dplyr 1.0.0, numerous improvements to SQL translation (including new Amazon Redshift and SAP HANA backends), and an improved system for extending dbplyr to work with other databases. [Read more ...](/blog/2020/11/dbplyr-2-0-0/) 2020/11/04 [![](/blog/2019/04/dbplyr-1-4-0/dbplyr-1-4-0-sq.jpg)](/blog/2019/04/dbplyr-1-4-0/) [dbplyr 1.4.0](/blog/2019/04/dbplyr-1-4-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2019/04/dbplyr-1-4-0/) 2019/04/30 [![](/blog/2019/02/bigrquery-1-1-0/bigrquery-1-1-0-sq.jpg)](/blog/2019/02/bigrquery-1-1-0/) [bigrquery 1.1.0](/blog/2019/02/bigrquery-1-1-0/) [package](/categories/package) Mara Averick bigrquery 1.1.0 is now on CRAN. [Read more ...](/blog/2019/02/bigrquery-1-1-0/) 2019/02/15 [![](/blog/2019/01/dbplyr-1-3-0/dbplyr-1-3-0-sq.jpg)](/blog/2019/01/dbplyr-1-3-0/) [dbplyr 1.3.0](/blog/2019/01/dbplyr-1-3-0/) [package](/categories/package) Mara Averick dbplyr 1.3.0 is now on CRAN. [Read more ...](/blog/2019/01/dbplyr-1-3-0/) 2019/01/09 [![](/blog/2018/04/bigrquery-1-0-0/bigrquery-1-0-0-sq.jpg)](/blog/2018/04/bigrquery-1-0-0/) [bigrquery 1.0.0](/blog/2018/04/bigrquery-1-0-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2018/04/bigrquery-1-0-0/) 2018/04/24 * [««](/tags/dbplyr/) * « * [1](/tags/dbplyr/) * [2](/tags/dbplyr/page/2/) * [»](/tags/dbplyr/page/2/) * [»»](/tags/dbplyr/page/2/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tune [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tune [![](/blog/2024/04/tune-1-2-0/thumbnail-sq.jpg)](/blog/2024/04/tune-1-2-0/) [tune 1.2.0](/blog/2024/04/tune-1-2-0/) [package](/categories/package) Simon Couch While we’ve written about survival analysis and machine learning fairness already, the newest tune release includes a number of other major changes. [Read more ...](/blog/2024/04/tune-1-2-0/) 2024/04/18 [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2024/03/tidymodels-fairness/thumbnail-sq.jpg)](/blog/2024/03/tidymodels-fairness/) [Fair machine learning with tidymodels](/blog/2024/03/tidymodels-fairness/) [learn](/categories/learn) Simon Couch Recent tidymodels releases integrated a set of tools for assessing whether machine learning models treat groups of people differently. [Read more ...](/blog/2024/03/tidymodels-fairness/) 2024/03/21 [![](/blog/2023/11/tidymodels-errors-q4/thumbnail-sq.jpg)](/blog/2023/11/tidymodels-errors-q4/) [Three ways errors are about to get better in tidymodels](/blog/2023/11/tidymodels-errors-q4/) [programming](/categories/programming) Simon Couch The tidymodels team’s biannual spring cleaning gave us a chance to revisit the way we raise some error messages. [Read more ...](/blog/2023/11/tidymodels-errors-q4/) 2023/11/10 [![](/blog/2023/08/validation-split-as-3-way-split/thumbnail-sq.jpg)](/blog/2023/08/validation-split-as-3-way-split/) [New interface to validation splits](/blog/2023/08/validation-split-as-3-way-split/) [package](/categories/package) Hannah Frick The latest releases of rsample and tune provide a new interface to validation sets as a three-way split. [Read more ...](/blog/2023/08/validation-split-as-3-way-split/) 2023/08/25 [![](/blog/2023/04/tuning-delights/thumbnail-sq.jpg)](/blog/2023/04/tuning-delights/) [Tuning hyperparameters with tidymodels is a delight](/blog/2023/04/tuning-delights/) [roundup](/categories/roundup) Simon Couch New releases of the tune, finetune, and workflowsets packages have made optimizing model parameters with tidymodels even more pleasant. [Read more ...](/blog/2023/04/tuning-delights/) 2023/04/20 [![](/blog/2021/07/tidymodels-july-2021/thumbnail-sq.jpg)](/blog/2021/07/tidymodels-july-2021/) [New tidymodels releases for July 2021](/blog/2021/07/tidymodels-july-2021/) [roundup](/categories/roundup) Max Kuhn We highlight a series of new tidymodels package versions and their improvements. [Read more ...](/blog/2021/07/tidymodels-july-2021/) 2021/07/27 [![](/blog/2020/12/finetune-0-0-1/thumbnail-sq.jpg)](/blog/2020/12/finetune-0-0-1/) [finetune 0.0.1](/blog/2020/12/finetune-0-0-1/) [package](/categories/package) Max Kuhn finetune is a new package that adds a few more model tuning methods. [Read more ...](/blog/2020/12/finetune-0-0-1/) 2020/12/02 [![](/blog/2020/11/tune-parallel/thumbnail-sq.jpg)](/blog/2020/11/tune-parallel/) [Parallel processing with tune](/blog/2020/11/tune-parallel/) [package](/categories/package) Max Kuhn With version 0.1.2 of tune, there are more options for parallel processing. [Read more ...](/blog/2020/11/tune-parallel/) 2020/11/27 [![](/blog/2020/11/tidymodels-sparse-support/thumbnail-sq.jpg)](/blog/2020/11/tidymodels-sparse-support/) [Sparse data structures in tidymodels](/blog/2020/11/tidymodels-sparse-support/) [learn](/categories/learn) Julia Silge Sparse data is common in many domains, and now tidymodels supports using sparse matrix structures throughout the fitting and tuning stages of modeling. [Read more ...](/blog/2020/11/tidymodels-sparse-support/) 2020/11/25 * [««](/tags/tune/) * « * [1](/tags/tune/) * [2](/tags/tune/page/2/) * [»](/tags/tune/page/2/) * [»»](/tags/tune/page/2/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q1 2024 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q1 2024 tidymodels digest ========================= ![](/blog/2024/04/tidymodels-2024-q1/thumbnail-wd.jpg) Photo by [Sergey Shmidt](https://unsplash.com/photos/orange-petaled-flowers-koy6FlCCy5s)   2024/04/24   [tidymodels](/tags/tidymodels/) , [workflows](/tags/workflows/) , [censored](/tags/censored/) , [workflowsets](/tags/workflowsets/)   Hannah Frick The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like these posts from the past couple of months: * [Survival analysis for time-to-event data with tidymodels](https://www.tidyverse.org/blog/2024/04/tidymodels-survival-analysis/) * [Fair machine learning with tidymodels](https://www.tidyverse.org/blog/2024/03/tidymodels-fairness/) * [tune 1.2.0](https://www.tidyverse.org/blog/2024/04/tune-1-2-0/) Additionally, we have published several related articles on [tidymodels.org](https://www.tidymodels.org/) : * [How long until building complaints are dispositioned? A survival analysis case study](https://www.tidymodels.org/learn/statistics/survival-case-study/) * [Dynamic Performance Metrics for Event Time Data](https://www.tidymodels.org/learn/statistics/survival-metrics/) * [Accounting for Censoring in Performance Metrics for Event Time Data](https://www.tidymodels.org/learn/statistics/survival-metrics-details/) * [Are GPT detectors fair? A machine learning fairness case study](https://www.tidymodels.org/learn/work/fairness-detectors/) * [Fair prediction of hospital readmission: a machine learning fairness case study](https://www.tidymodels.org/learn/work/fairness-readmission/) * [Confidence Intervals for Performance Metrics](https://www.tidymodels.org/learn/models/bootstrap-metrics/) Since [our last roundup post](https://www.tidyverse.org/blog/2024/01/tidymodels-2023-q4/) , there have been CRAN releases of 21 tidymodels packages. Here are links to their NEWS files: * baguette [(1.0.2)](https://baguette.tidymodels.org/news/index.html) * brulee [(0.3.0)](https://brulee.tidymodels.org/news/index.html) * butcher [(0.3.4)](https://butcher.tidymodels.org/news/index.html) * censored [(0.3.0)](https://censored.tidymodels.org/news/index.html) * dials [(1.2.1)](https://dials.tidymodels.org/news/index.html) * embed [(1.1.4)](https://embed.tidymodels.org/news/index.html) * finetune [(1.2.0)](https://finetune.tidymodels.org/news/index.html) * hardhat [(1.3.1)](https://hardhat.tidymodels.org/news/index.html) * modeldata [(1.3.0)](https://modeldata.tidymodels.org/news/index.html) * parsnip [(1.2.1)](https://parsnip.tidymodels.org/news/index.html) * probably [(1.0.3)](https://probably.tidymodels.org/news/index.html) * recipes [(1.0.10)](https://recipes.tidymodels.org/news/index.html) * rsample [(1.2.1)](https://rsample.tidymodels.org/news/index.html) * shinymodels [(0.1.1)](https://shinymodels.tidymodels.org/news/index.html) * stacks [(1.0.4)](https://stacks.tidymodels.org/news/index.html) * tidyclust [(0.2.1)](https://tidyclust.tidymodels.org/news/index.html) * tidymodels [(1.2.0)](https://tidymodels.tidymodels.org/news/index.html) * tune [(1.2.0)](https://tune.tidymodels.org/news/index.html) * workflows [(1.1.4)](https://workflows.tidymodels.org/news/index.html) * workflowsets [(1.1.0)](https://workflowsets.tidymodels.org/news/index.html) * yardstick [(1.3.1)](https://yardstick.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: new prediction options in censored, consistency in augmenting parsnip models and workflows, as well as a new autoplot type for workflow sets. library(tidymodels) library(censored) New prediction options in censored[](#new-prediction-options-in-censored) -------------------------------------------------------------------------- As part of the framework-wide integration of survival analysis, the parsnip extension package censored has received some love in the form of new prediction options. Random forests with the `"aorsf"` engine can now predict survival time, thanks to the new feature in the [aorsf](https://docs.ropensci.org/aorsf/) package itself. This means that all engines in censored can now predict survival time. Let’s predict survival time for the first five rows of the lung cancer dataset, survival analysis’ `mtcars`. rf_spec <- rand_forest() |> set_engine("aorsf") |> set_mode("censored regression") rf_fit <- rf_spec |> fit(Surv(time, status) ~ age + sex, data = lung) lung_5 <- lung[1:5, ] predict(rf_fit, new_data = lung_5, type = "time") #> # A tibble: 5 × 1 #> .pred_time #> #> 1 217. #> 2 240. #> 3 236. #> 4 236. #> 5 254. Some models allow for predictions based on different values for tuning parameter without having to refit the model. In parsnip, we refer to this as [“the submodel trick."](https://parsnip.tidymodels.org/articles/Submodels.html) Some of those models are regularized models fitted with the [glmnet](https://glmnet.stanford.edu/) engine. In censored, the corresponding [`multi_predict()`](https://parsnip.tidymodels.org/reference/multi_predict.html) method has now gained the prediction types `"time"` and `"raw"` in addition to the existing types `"survival"` and `"linear_pred"`. Let’s fit a regularized Cox model to illustrate. Note how we set the `penalty` to a fixed value of `0.1`. cox_fit <- proportional_hazards(penalty = 0.1) |> set_engine("glmnet") |> set_mode("censored regression") |> fit(Surv(time, status) ~ ., data = lung) Predictions made with [`predict()`](https://rdrr.io/r/stats/predict.html) use that penalty value of 0.1. With [`multi_predict()`](https://parsnip.tidymodels.org/reference/multi_predict.html) , we can change that value to something different without having to refit. Conveniently, we can predict for multiple penalty values as well. predict(cox_fit, new_data = lung_5, type = "time") #> # A tibble: 5 × 1 #> .pred_time #> #> 1 NA #> 2 425. #> 3 NA #> 4 350. #> 5 NA mpred <- multi_predict(cox_fit, new_data = lung_5, type = "time", penalty = c(0.01, 0.1)) mpred #> # A tibble: 5 × 1 #> .pred #> #> 1 #> 2 #> 3 #> 4 #> 5 The resulting tibble is nested by observation to follow the convention of one row per observation. For each observation, the predictions are stored in a tibble containing the penalty value along with the prediction. mpred$.pred[[2]] #> # A tibble: 2 × 2 #> penalty .pred_time #> #> 1 0.01 461. #> 2 0.1 425. You can see that the predicted value from [`predict()`](https://rdrr.io/r/stats/predict.html) matches the predicted value from [`multi_predict()`](https://parsnip.tidymodels.org/reference/multi_predict.html) with a penalty of 0.1. Consistent `augment()` for workflows and parsnip models[](#consistent-augment-for-workflows-and-parsnip-models) ---------------------------------------------------------------------------------------------------------------- If you are interested in exploring predictions in relation to predictors, [`augment()`](https://generics.r-lib.org/reference/augment.html) is your extended [`predict()`](https://rdrr.io/r/stats/predict.html) method: it will augment the inputted dataset with its predictions. For classification, it will add hard class predictions as well as class probabilities. For regression, it will add the numeric prediction. If the outcome variable is part of the dataset, it also calculates residuals. This has already been the case for fitted parsnip models, and the [`augment()`](https://generics.r-lib.org/reference/augment.html) method for workflows will now also calculate residuals. spec_fit <- fit(linear_reg(), mpg ~ ., mtcars) wflow_fit <- workflow(mpg ~ ., linear_reg()) %>% fit(mtcars) augment(spec_fit, mtcars) #> # A tibble: 32 × 13 #> .pred .resid mpg cyl disp hp drat wt qsec vs am gear #> #> 1 22.6 -1.60 21 6 160 110 3.9 2.62 16.5 0 1 4 #> 2 22.1 -1.11 21 6 160 110 3.9 2.88 17.0 0 1 4 #> 3 26.3 -3.45 22.8 4 108 93 3.85 2.32 18.6 1 1 4 #> 4 21.2 0.163 21.4 6 258 110 3.08 3.22 19.4 1 0 3 #> 5 17.7 1.01 18.7 8 360 175 3.15 3.44 17.0 0 0 3 #> 6 20.4 -2.28 18.1 6 225 105 2.76 3.46 20.2 1 0 3 #> 7 14.4 -0.0863 14.3 8 360 245 3.21 3.57 15.8 0 0 3 #> 8 22.5 1.90 24.4 4 147. 62 3.69 3.19 20 1 0 4 #> 9 24.4 -1.62 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 #> 10 18.7 0.501 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 #> # ℹ 22 more rows #> # ℹ 1 more variable: carb augment(wflow_fit, mtcars) #> # A tibble: 32 × 13 #> .pred .resid mpg cyl disp hp drat wt qsec vs am gear #> * #> 1 22.6 -1.60 21 6 160 110 3.9 2.62 16.5 0 1 4 #> 2 22.1 -1.11 21 6 160 110 3.9 2.88 17.0 0 1 4 #> 3 26.3 -3.45 22.8 4 108 93 3.85 2.32 18.6 1 1 4 #> 4 21.2 0.163 21.4 6 258 110 3.08 3.22 19.4 1 0 3 #> 5 17.7 1.01 18.7 8 360 175 3.15 3.44 17.0 0 0 3 #> 6 20.4 -2.28 18.1 6 225 105 2.76 3.46 20.2 1 0 3 #> 7 14.4 -0.0863 14.3 8 360 245 3.21 3.57 15.8 0 0 3 #> 8 22.5 1.90 24.4 4 147. 62 3.69 3.19 20 1 0 4 #> 9 24.4 -1.62 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 #> 10 18.7 0.501 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 #> # ℹ 22 more rows #> # ℹ 1 more variable: carb Both methods also append on the left-hand side of the data frame, rather than the right-hand side. This means that prediction columns are always visible when printed, even for data frames with many columns. As you might expect, the order of the columns is the same for both methods as well. New autoplot type for workflow sets[](#new-autoplot-type-for-workflow-sets) ---------------------------------------------------------------------------- Many tidymodels objects have [`autoplot()`](https://ggplot2.tidyverse.org/reference/autoplot.html) methods for quickly getting a sense of the most important aspects of an object. For workflow sets, the method shows the value of the calculated performance metrics, as well as the respective rank of each workflow in the set. Let’s put together a workflow set on the actual `mtcars` data and take a look at the default autoplot. mt_rec <- recipe(mpg ~ ., mtcars) mt_rec2 <- mt_rec |> step_normalize(all_numeric_predictors()) mt_rec3 <- mt_rec |> step_YeoJohnson(all_numeric_predictors()) wflow_set <- workflow_set( list(plain = mt_rec, normalize = mt_rec2, yeo_johnson = mt_rec3), list(linear_reg()) ) set.seed(1) wflow_set_fit <- workflow_map( wflow_set, "fit_resamples", resamples = bootstraps(mtcars) ) autoplot(wflow_set_fit) ![](figs/workflowsets-autoplot-1.png) This allows you to grasp the metric values and rank of a workflow and let’s you distinguish the type of preprocessor and model. In our case, we only have one type of model, and even just one type of preprocessor, a recipe. What we are much more interested in is which recipe corresponds to which rank. The new option of `type = "wflow_id"` lets us see which values and ranks correspond with which workflow and thus also with which recipe. autoplot(wflow_set_fit, type = "wflow_id") ![](figs/workflowsets-autoplot-new-1.png) This makes it easy to spot that it’s the Yeo-Johnson transformation that makes the difference here! Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to thank those in the community that contributed to tidymodels in the last quarter: * baguette: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@topepo](https://github.com/topepo) . * brulee: [@jrosell](https://github.com/jrosell) , and [@topepo](https://github.com/topepo) . * butcher: [@juliasilge](https://github.com/juliasilge) . * censored: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@simonpcouch](https://github.com/simonpcouch) , and [@tripartio](https://github.com/tripartio) . * dials: [@hfrick](https://github.com/hfrick) , and [@topepo](https://github.com/topepo) . * embed: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) . * finetune: [@hfrick](https://github.com/hfrick) , [@jrosell](https://github.com/jrosell) , [@mfansler](https://github.com/mfansler) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * hardhat: [@DavisVaughan](https://github.com/DavisVaughan) , and [@simonpcouch](https://github.com/simonpcouch) . * modeldata: [@topepo](https://github.com/topepo) . * parsnip: [@birbritto](https://github.com/birbritto) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@jmunyoon](https://github.com/jmunyoon) , [@marcelglueck](https://github.com/marcelglueck) , [@mattheaphy](https://github.com/mattheaphy) , [@mesdi](https://github.com/mesdi) , [@nipnipj](https://github.com/nipnipj) , [@pgg1309](https://github.com/pgg1309) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@wzbillings](https://github.com/wzbillings) . * probably: [@brshallo](https://github.com/brshallo) , [@Jeffrothschild](https://github.com/Jeffrothschild) , [@jgaeb](https://github.com/jgaeb) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * recipes: [@DemetriPananos](https://github.com/DemetriPananos) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@jdonland](https://github.com/jdonland) , [@JiahuaQu](https://github.com/JiahuaQu) , [@joranE](https://github.com/joranE) , [@mikemahoney218](https://github.com/mikemahoney218) , [@olivroy](https://github.com/olivroy) , [@SantiagoD999](https://github.com/SantiagoD999) , [@simonpcouch](https://github.com/simonpcouch) , [@stufield](https://github.com/stufield) , and [@topepo](https://github.com/topepo) . * rsample: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@mikemahoney218](https://github.com/mikemahoney218) , [@paulcbauer](https://github.com/paulcbauer) , [@StevenWallaert](https://github.com/StevenWallaert) , [@topepo](https://github.com/topepo) , and [@ZWael](https://github.com/ZWael) . * shinymodels: [@simonpcouch](https://github.com/simonpcouch) . * stacks: [@simonpcouch](https://github.com/simonpcouch) . * tidyclust: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@katieburak](https://github.com/katieburak) . * tidymodels: [@jkylearmstrong](https://github.com/jkylearmstrong) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@nikosGeography](https://github.com/nikosGeography) , [@nipnipj](https://github.com/nipnipj) , and [@topepo](https://github.com/topepo) . * tune: [@AlbertoImg](https://github.com/AlbertoImg) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@joranE](https://github.com/joranE) , [@joshuagi](https://github.com/joshuagi) , [@lionel-](https://github.com/lionel-) , [@marcozanotti](https://github.com/marcozanotti) , [@Peter4801](https://github.com/Peter4801) , [@rfsaldanha](https://github.com/rfsaldanha) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@walkerjameschris](https://github.com/walkerjameschris) . * workflows: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@mesdi](https://github.com/mesdi) , [@Milardkh](https://github.com/Milardkh) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * workflowsets: [@hfrick](https://github.com/hfrick) , and [@simonpcouch](https://github.com/simonpcouch) . * yardstick: [@asb2111](https://github.com/asb2111) , [@Dpananos](https://github.com/Dpananos) , [@EduMinsky](https://github.com/EduMinsky) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , and [@tripartio](https://github.com/tripartio) . We’re grateful for all of the tidymodels community, from observers to users to contributors. Happy modeling! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # marquee 0.1.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) marquee 0.1.0 ============= ![](/blog/2024/05/marquee-0-1-0/thumbnail-wd.jpg) Photo by [Etienne Girardet](https://unsplash.com/photos/a-close-up-of-a-metal-grate-on-a-table-9jfpVAhGC1g)   2024/05/29   [marquee](/tags/marquee/) , [ggplot2](/tags/ggplot2/)   Thomas Lin Pedersen I am super excited to announce the initial release of [marquee](https://marquee.r-lib.org) , a markdown parser and renderer for R graphics that allows native rich text formatting of text in graphics created with grid (which includes ggplot2 and lattice). The inception of this package goes all the way back to 2017: > May I present: Text wrapping of theme elements in [#ggplot2](https://twitter.com/hashtag/ggplot2?src=hash&ref_src=twsrc%5Etfw) > with the new (experimental) element\_textbox in [#ggforce](https://twitter.com/hashtag/ggforce?src=hash&ref_src=twsrc%5Etfw) > [#rstats](https://twitter.com/hashtag/rstats?src=hash&ref_src=twsrc%5Etfw) > [#dataviz](https://twitter.com/hashtag/dataviz?src=hash&ref_src=twsrc%5Etfw) > [pic.twitter.com/JJMLcuTBqx](https://t.co/JJMLcuTBqx) > > — Thomas Lin Pedersen (@thomasp85) [January 5, 2017](https://twitter.com/thomasp85/status/816967301014634497?ref_src=twsrc%5Etfw) (yeah…) where I developed an experimental feature for ggforce that allowed automatic text wrapping in [`element_text()`](https://ggplot2.tidyverse.org/reference/element.html) . Years passed, slowly improving the text rendering capabilities in R until we are finally at a point in the toolchain where something like marquee can deliver on my initial plans. If this has you intrigued you can install it from CRAN with: install.packages("marquee") This blog post will go through the features of marquee, along with discussing some of its current limitations, all of which are hopefully transient. library(marquee) An example[](#an-example) -------------------------- Since the use of markdown is second-hand nature for most people at this point, there shouldn’t be much surprise in what marquee is capable off, so let’s start with an example to show the main use: md_text <- "# Intro markdown has been *quite* succesful in creating a unified way of specifying _semantic_ rich text. While limited, it provides both {.steelblue readability} and just enough ~power~ features. text <- \"markdown **text**\" marquee_grob(text) It features, among others: 1. lists 2. code blocks * Indented lists 3. and more... " grid::grid.draw(marquee_grob(md_text)) ![](figs/unnamed-chunk-3-1.png) The above illustrates a couple of things. First and foremost, that markdown works in very unsurprising ways and you get what you type. In fact, the full CommonMark syntax is supported along with extensions for underline and strikethrough. Further, it shows that marquee provides its own extension for specifying custom span elements in the form of the `{.class }` syntax. The renderer is clever in interpreting the class so that if it corresponds to a colour name, the colour is automatically applied to the text. Lastly, it shows that the default styling of markdown closely follows the look you’ve come to expect from markdown rendered to HTML. Use in ggplot2[](#use-in-ggplot2) ---------------------------------- The number of people using this directly in grid is probably small. It is more likely that you access the functionality of marquee through higher level functions. Marquee provides two such functions aimed at making it easy to use marquee in ggplot2. The aim is to eventually move these into ggplot2 proper, but while we are in the initial phase of development they will stay in this package. ### `geom_marquee()`[](#geom_marquee) The first function is (obviously) a geom. It is intended as a stand-in replacement for both [`geom_text()`](https://ggplot2.tidyverse.org/reference/geom_text.html) and [`geom_label()`](https://ggplot2.tidyverse.org/reference/geom_text.html) . As with [`marquee_grob()`](https://marquee.r-lib.org/reference/marquee_grob.html) it works very unsurprisingly: library(ggplot2) # Add styling around the first word red_bold_names <- sub("(\\w+)", "{.red **\\1**}", rownames(mtcars)) ggplot(mtcars) + geom_marquee(aes(x = mpg, y = disp, label = red_bold_names)) ![](figs/unnamed-chunk-4-1.png) Apart from standard, but markdown-aware, [`geom_text()`](https://ggplot2.tidyverse.org/reference/geom_text.html) behaviour, the geom also gains a `width` aesthetic that allows you to turn on automatic soft wrapping of the text. In addition to this it gains a `style` aesthetic to finely control the style (more about styling below) ### `element_marquee()`[](#element_marquee) The second obvious use for marquee in ggplot2 is in formatting text elements. [`element_marquee()`](https://marquee.r-lib.org/reference/element_marquee.html) is a replacement for [`element_text()`](https://ggplot2.tidyverse.org/reference/element.html) that does just that. ggplot(mtcars) + geom_point(aes(x = mpg, y = disp)) + ggtitle(md_text) + theme(plot.title = element_marquee(size = 8, width = unit(16, "cm"))) ![](figs/unnamed-chunk-5-1.png) Styling[](#styling) -------------------- As alluded to above, marquee comes with a styling API that is reminiscent of CSS but completely its own. In some sense it takes the “simplicity over power” approach from markdown and applies it to styling. In marquee, each element type (e.g. a code block) has its own style. This style can be incomplete in which case it inherits the remaining specifications from the parent element in the document. As an example, the `em` element has the following default style `style(italic = TRUE)`, that is, take whatever style is currently in effect but also make the text italic. Apart from the direct inheritance of the marquee styling, it is also possible to use relative inheritance for numeric specifications (e.g. `lineheight = relative(2)` to double the current lineheight) or set sizes based on the current or root element font size (using [`em()`](https://marquee.r-lib.org/reference/style_helpers.html) and [`rem()`](https://marquee.r-lib.org/reference/style_helpers.html) respectively). Lastly, you can also mark a specification as “non-inheritable” using [`skip_inherit()`](https://marquee.r-lib.org/reference/style_helpers.html) . This essentially instructs any children to not inherit the value but instead inherit the value from the grand-parent element. Images[](#images) ------------------ Markdown (famously) supports adding images through the `![alt text](path/to/image)` syntax. Since marquee supports the full CommonMark spec, this is of course also supported. The only limitation is that the “alt text” is ignored since hovering tool-tips or screen-readers are not supported for the output types that marquee renders to. If an image is placed on a line together with surrounding text it will be rendered to fit the line height of the line. If it is placed by itself on its own line it will span the width available: logo <- system.file("help", "figures", "logo.png", package = "marquee") header_img <- "thumbnail-wd.jpg" md_img <- "# About marquee ![]({logo}) Both PNG (above), JPEG (below), and SVG (not shown) are supported ![]({header_img}) The above image is treated like a block element " md_img <- marquee_glue(md_img) grid::grid.draw(marquee_grob(md_img)) ![](figs/unnamed-chunk-6-1.png) Apart from showing support for images we also introduce a new function above, [`marquee_glue()`](https://marquee.r-lib.org/reference/marquee_glue.html) . It is a function that works very much like [`glue::glue()`](https://glue.tidyverse.org/reference/glue.html) and performs text interpolation. However, this variant understands the custom span syntax of marquee so that these will not be treated as interpolation sites. Further, it turns off the `#` interpretation as a comment character as this interferes with the markdown header syntax. All of the above is pretty standard markdown and since I prefixed this whole blog post with “full markdown support” it shouldn’t come as a big surprise. However, marquee has one last trick up its sleeve: R graphics interpolation. Quite simply, if you, instead of providing a path to a file, provide the name of an R variable holding a graphic object, this will be included as an image. Here’s how it works: plot <- ggplot(mtcars) + geom_point(aes(mpg, disp)) + geom_point(aes(mpg, disp), mtcars[1,], colour = "red", size = 3) point <- grid::pointsGrob(x = 0.5, y = 0.5, pch = 19, gp = grid::gpar(col = "red")) md_plots <- "# Plots In the plot below, the red dot (![](point)) shows the Mazda RX4 ![](plot) " grid::grid.draw(marquee_grob(md_plots)) ![](figs/unnamed-chunk-7-1.png) This also means that your ggplots can contain additional ggplots (or other graphics) anywhere you are allowed to place text (using [`geom_marquee()`](https://marquee.r-lib.org/reference/geom_marquee.html) and [`element_marquee()`](https://marquee.r-lib.org/reference/element_marquee.html) ) - for better or worse… ![](figs/they_didnt_stop.gif) Limitations[](#limitations) ---------------------------- Marquee’s biggest limitation is its reliance on very new features in the graphics engine. The rendering will _not_ work on anything before R 4.3, but even then it requires the graphics device to support a range of new features, most importantly the new glyph specification introduced in R 4.3. While several graphics devices do support the required features, most notably those powered by Cairo as well as all devices in ragg, many do not. The default Windows graphics device continues to lag behind and the default on macOS, while supporting glyphs, can crash in some situations bringing the whole R session down with it (this is still being investigated). So we are in no doubt threading the frontier here. All of this is set to resolve itself (maybe except for the default Windows device) as time passes. A limitation of great interest to me is the lack of support in svglite. svglite is build on a core idea of post-editability and thus wants all its text to be selectable and editable when opened in a capable program such as Adobe Illustrator. However, the graphics engine API that powers the new capabilities does not really allow this and I’m still figuring out how to reconcile it. It will eventually be solved though. Lastly, while not really part of HTML syntax directly, many people rely on HTML inside markdown documents to solve layout and styling tasks that markdown doesn’t support. The way it works is that markdown passes the HTML through unmodified and then the HTML is parsed by the HTML renderer (often the browser) used to display the rendered markdown document. This makes it seem like understanding HTML is part of markdown, while it’s really not. The reason I’m going through all this explanation is to say that marquee has no understanding of HTML and will not render it as expected. While some HTML tags and CSS settings have clear counterparts in markdown and the marquee styling system it is much better to have a clear “no-support” over an arbitrary limited support. [`marquee_grob()`](https://marquee.r-lib.org/reference/marquee_grob.html) / [`marquee_parse()`](https://marquee.r-lib.org/reference/marquee_parse.html) have an argument (`ignore_html`) that controls whether HTML are outright removed from the output (default), or if it is included verbatim. Acknowledgements[](#acknowledgements) -------------------------------------- Marquee is the latest in a stream of advancements when it comes to text rendering and font support in R. It builds on top of my work with [systemfonts](https://systemfonts.r-lib.org/) , [textshaping](https://github.com/r-lib/textshaping) , and [ragg](https://ragg.r-lib.org/) , but also pays great debt to Paul Murrell’s work on adding a new, more low level API for text rendering to grid and the graphics engine. Lastly, Claus Wilke’s work on [gridtext](https://wilkelab.org/gridtext/) and [ggtext](https://wilkelab.org/ggtext/) showed the power and need for rich text support in R and filled a gap until the technical foundation for marquee was built out. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tune 1.2.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) tune 1.2.0 ========== ![](/blog/2024/04/tune-1-2-0/thumbnail-wd.jpg) Photo by [Derek Story](https://unsplash.com/photos/1Pzhr6XPl6k)   2024/04/18   [tidymodels](/tags/tidymodels/) , [parallelism](/tags/parallelism/) , [tune](/tags/tune/)   Simon Couch We’re indubitably amped to announce the release of [tune](https://tune.tidymodels.org/) 1.2.0, a package for hyperparameter tuning in the [tidymodels framework](https://www.tidymodels.org/) . You can install it from CRAN, along with the rest of the core packages in tidymodels, using the tidymodels meta-package: install.packages("tidymodels") The 1.2.0 release of tune has introduced support for two major features that we’ve written about on the tidyverse blog already: * [Survival analysis for time-to-event data with tidymodels](https://www.tidyverse.org/blog/2024/04/tidymodels-survival-analysis/) * [Fair machine learning with tidymodels](https://www.tidyverse.org/blog/2024/03/tidymodels-fairness/) While those features got their own blog posts, there are several more features in this release that we thought were worth calling out. This post will highlight improvements to our support for parallel processing, the introduction of support for percentile confidence intervals for performance metrics, and a few other bits and bobs. You can see a full list of changes in the [release notes](https://github.com/tidymodels/tune/releases/tag/v1.2.0) . library(tidymodels) Throughout this post, I’ll refer to the example of tuning an XGBoost model to predict the fuel efficiency of various car models. I hear this is already a well-explored modeling problem, but alas: set.seed(2024) xgb_res <- tune_grid( boost_tree(mode = "regression", mtry = tune(), learn_rate = tune()), mpg ~ ., bootstraps(mtcars), control = control_grid(save_pred = TRUE) ) Note that we’ve used the [control option](https://tune.tidymodels.org/reference/control_grid.html) `save_pred = TRUE` to indicate that we want to save the predictions from our resampled models in the tuning results. Both `int_pctl()` and `compute_metrics()` below will need those predictions. The metrics for our resampled model look like so: collect_metrics(xgb_res) #> # A tibble: 20 × 8 #> mtry learn_rate .metric .estimator mean n std_err .config #> #> 1 2 0.00204 rmse standard 19.7 25 0.262 Preprocessor1_Model01 #> 2 2 0.00204 rsq standard 0.659 25 0.0314 Preprocessor1_Model01 #> 3 6 0.00859 rmse standard 18.0 25 0.260 Preprocessor1_Model02 #> 4 6 0.00859 rsq standard 0.607 25 0.0270 Preprocessor1_Model02 #> 5 3 0.0276 rmse standard 14.0 25 0.267 Preprocessor1_Model03 #> 6 3 0.0276 rsq standard 0.710 25 0.0237 Preprocessor1_Model03 #> # ℹ 14 more rows Modernized support for parallel processing[](#modernized-support-for-parallel-processing) ------------------------------------------------------------------------------------------ The tidymodels framework has long supported evaluating models in parallel using the [foreach](https://cran.r-project.org/web/packages/foreach/vignettes/foreach.html) package. This release of tune has introduced support for parallelism using the [futureverse](https://www.futureverse.org/) framework, and we will begin deprecating our support for foreach in a coming release. To tune a model in parallel with foreach, a user would load a _parallel backend_ package (usually with a name like [`library(doBackend)`](https://rdrr.io/r/base/library.html) ) and then _register_ it with foreach (with a function call like `registerDoBackend()`). The tune package would then detect that registered backend and take it from there. For example, the code to distribute the above tuning process across 10 cores with foreach would look like: library(doMC) registerDoMC(cores = 10) set.seed(2024) xgb_res <- tune_grid( boost_tree(mode = "regression", mtry = tune(), learn_rate = tune()), mpg ~ ., bootstraps(mtcars), control = control_grid(save_pred = TRUE) ) The code to do so with future is similarly simple. Users first load the [future](https://future.futureverse.org/) package, and then specify a [`plan()`](https://future.futureverse.org/reference/plan.html) which dictates how computations will be distributed. For example, the code to distribute the above tuning process across 10 cores with future looks like: library(future) plan(multisession, workers = 10) set.seed(2024) xgb_res <- tune_grid( boost_tree(mode = "regression", mtry = tune(), learn_rate = tune()), mpg ~ ., bootstraps(mtcars), control = control_grid(save_pred = TRUE) ) For users, the transition to parallelism with future has several benefits: * The futureverse presently supports a greater number of parallelism technologies and has been more likely to receive implementations for new ones. * Once foreach is fully deprecated, users will be able to use the [interactive logger](https://www.tidyverse.org/blog/2023/04/tuning-delights/#interactive-issue-logging) when tuning in parallel. From our perspective, transitioning our parallelism support to future makes our packages much more maintainable, reducing complexity in random number generation, error handling, and progress reporting. In an upcoming release of the package, you’ll see a deprecation warning when a foreach parallel backend is registered but no future plan has been specified, so start transitioning your code sooner than later! Percentile confidence intervals[](#percentile-confidence-intervals) -------------------------------------------------------------------- Following up on changes in the [most recent rsample release](https://github.com/tidymodels/rsample/releases/tag/v1.2.0) , tune has introduced a [method for `int_pctl()`](https://tune.tidymodels.org/reference/int_pctl.tune_results.html) that calculates percentile confidence intervals for performance metrics. To calculate a 90% confidence interval for the values of each performance metric returned in `collect_metrics()`, we’d write: set.seed(2024) int_pctl(xgb_res, alpha = .1) #> # A tibble: 20 × 8 #> .metric .estimator .lower .estimate .upper .config mtry learn_rate #> #> 1 rmse bootstrap 18.1 19.9 22.0 Preprocessor1_Mod… 2 0.00204 #> 2 rsq bootstrap 0.570 0.679 0.778 Preprocessor1_Mod… 2 0.00204 #> 3 rmse bootstrap 16.6 18.3 19.9 Preprocessor1_Mod… 6 0.00859 #> 4 rsq bootstrap 0.548 0.665 0.765 Preprocessor1_Mod… 6 0.00859 #> 5 rmse bootstrap 12.5 14.1 15.9 Preprocessor1_Mod… 3 0.0276 #> 6 rsq bootstrap 0.622 0.720 0.818 Preprocessor1_Mod… 3 0.0276 #> # ℹ 14 more rows Note that the output has the same number of rows as the `collect_metrics()` output: one for each unique pair of metric and workflow. This is very helpful for validation sets. Other resampling methods generate replicated performance statistics. We can compute simple interval estimates using the mean and standard error for those. Validation sets produce only one estimate, and these bootstrap methods are probably the best option for obtaining interval estimates. Breaking change: relocation of ellipses[](#breaking-change-relocation-of-ellipses) ----------------------------------------------------------------------------------- We’ve made a **breaking change** in argument order for several functions in the package (and downstream packages like finetune and workflowsets). Ellipses (…) are now used consistently in the package to require optional arguments to be named. For functions that previously had unused ellipses at the end of the function signature, they have been moved to follow the last argument without a default value, and several other functions that previously did not have ellipses in their signatures gained them. This applies to methods for `augment()`, `collect_predictions()`, `collect_metrics()`, `select_best()`, `show_best()`, and `conf_mat_resampled()`. Compute new metrics without re-fitting[](#compute-new-metrics-without-re-fitting) ---------------------------------------------------------------------------------- We’ve also added a new function, [`compute_metrics()`](https://tune.tidymodels.org/reference/compute_metrics.html) , that allows for calculating metrics that were not used when evaluating against resamples. For example, consider our `xgb_res` object. Since we didn’t supply any metrics to evaluate, and this model is a regression model, tidymodels selected RMSE and R2 as defaults: collect_metrics(xgb_res) #> # A tibble: 20 × 8 #> mtry learn_rate .metric .estimator mean n std_err .config #> #> 1 2 0.00204 rmse standard 19.7 25 0.262 Preprocessor1_Model01 #> 2 2 0.00204 rsq standard 0.659 25 0.0314 Preprocessor1_Model01 #> 3 6 0.00859 rmse standard 18.0 25 0.260 Preprocessor1_Model02 #> 4 6 0.00859 rsq standard 0.607 25 0.0270 Preprocessor1_Model02 #> 5 3 0.0276 rmse standard 14.0 25 0.267 Preprocessor1_Model03 #> 6 3 0.0276 rsq standard 0.710 25 0.0237 Preprocessor1_Model03 #> # ℹ 14 more rows In the past, if you wanted to evaluate that workflow against a performance metric that you hadn’t included in your `tune_grid()` run, you’d need to re-run `tune_grid()`, fitting models and predicting new values all over again. Now, using the `compute_metrics()` function, you can use the `tune_grid()` output you’ve already generated and compute any number of new metrics without having to fit any more models as long as you use the control option `save_pred = TRUE` when tuning. So, say I want to additionally calculate Huber Loss and Mean Absolute Percent Error. I just pass those metrics along with the tuning result to `compute_metrics()`, and the result looks just like `collect_metrics()` output for the metrics originally calculated: compute_metrics(xgb_res, metric_set(huber_loss, mape)) #> # A tibble: 20 × 8 #> mtry learn_rate .metric .estimator mean n std_err .config #> #> 1 2 0.00204 huber_loss standard 18.3 25 0.232 Preprocessor1_Mode… #> 2 2 0.00204 mape standard 94.4 25 0.0685 Preprocessor1_Mode… #> 3 6 0.00859 huber_loss standard 16.7 25 0.229 Preprocessor1_Mode… #> 4 6 0.00859 mape standard 85.7 25 0.178 Preprocessor1_Mode… #> 5 3 0.0276 huber_loss standard 12.6 25 0.230 Preprocessor1_Mode… #> 6 3 0.0276 mape standard 64.4 25 0.435 Preprocessor1_Mode… #> # ℹ 14 more rows Easily pivot resampled metrics[](#easily-pivot-resampled-metrics) ------------------------------------------------------------------ Finally, the `collect_metrics()` method for tune results recently [gained a new argument](https://tune.tidymodels.org/reference/collect_predictions.html#arguments) , `type`, indicating the shape of the returned metrics. The default, `type = "long"`, is the same shape as before. The argument value `type = "wide"` will allot each metric its own column, making it easier to compare metrics across different models. collect_metrics(xgb_res, type = "wide") #> # A tibble: 10 × 5 #> mtry learn_rate .config rmse rsq #> #> 1 2 0.00204 Preprocessor1_Model01 19.7 0.659 #> 2 6 0.00859 Preprocessor1_Model02 18.0 0.607 #> 3 3 0.0276 Preprocessor1_Model03 14.0 0.710 #> 4 2 0.0371 Preprocessor1_Model04 12.3 0.728 #> 5 5 0.00539 Preprocessor1_Model05 18.8 0.595 #> 6 9 0.0110 Preprocessor1_Model06 17.4 0.577 #> # ℹ 4 more rows Under the hood, this is indeed just a `pivot_wider()` call. We’ve found that it’s time-consuming and error-prone to programmatically determine identifying columns when pivoting resampled metrics, so we’ve localized and thoroughly tested the code that we use to do so with this feature. More love for the Brier score[](#more-love-for-the-brier-score) ---------------------------------------------------------------- Tuning and resampling functions use default metrics when the user does not specify a custom metric set. For regression models, these are RMSE and R2. For classification, accuracy and the area under the ROC curve _were_ the default. We’ve also added the [Brier score](https://en.wikipedia.org/wiki/Brier_score) to the default classification metric list. Acknowledgements[](#acknowledgements) -------------------------------------- As always, we’re appreciative of the community contributors who helped make this release happen: [@AlbertoImg](https://github.com/AlbertoImg) , [@dramanica](https://github.com/dramanica) , [@epiheather](https://github.com/epiheather) , [@joranE](https://github.com/joranE) , [@jrosell](https://github.com/jrosell) , [@jxu](https://github.com/jxu) , [@kbodwin](https://github.com/kbodwin) , [@kenraywilliams](https://github.com/kenraywilliams) , [@KJT-Habitat](https://github.com/KJT-Habitat) , [@lionel-](https://github.com/lionel-) , [@marcozanotti](https://github.com/marcozanotti) , [@MasterLuke84](https://github.com/MasterLuke84) , [@mikemahoney218](https://github.com/mikemahoney218) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , and [@Peter4801](https://github.com/Peter4801) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dbplyr 2.5.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dbplyr 2.5.0 ============ ![](/blog/2024/04/dbplyr-2-5-0/thumbnail-wd.jpg) Photo by [jesse orrico](https://unsplash.com/photos/gray-metal-drawers-h6xNSDlgciU)   2024/04/08   [dbplyr](/tags/dbplyr/)   Hadley Wickham We’re most pleased to announce the release of [dbplyr](http://dbplyr.tidyverse.org/) 2.5.0. dbplyr is a database backend for dplyr that allows you to use a remote database as if it was a collection of local data frames: you write ordinary dplyr code and dbplyr translates it to SQL for you. You can install it from CRAN with: install.packages("dbplyr") This post focuses on the biggest change in dbplyr 2.5.0: improved syntax for tables nested inside schema and catalogs. As usual, this release also contains a ton of minor improvements to SQL generation, and I’d highly recommend skimming the [release notes](https://github.com/tidyverse/dbplyr/releases/tag/v2.5.0) to learn the details. library(dbplyr) Referring to tables in a schema[](#referring-to-tables-in-a-schema) -------------------------------------------------------------------- Historically, dbplyr has provided a bewildering array of options to specify a table inside a schema inside a catalog: con |> tbl(ident_q("catalog_name.schema_name.table_name")) con |> tbl(sql("SELECT * FROM catalog_name.schema_name.table_name")) con |> tbl(in_catalog("catalog_name", "schema_name", "table_name")) con |> tbl(ident_q("catalog_name.schema_name"), "table_name") con |> tbl(sql("catalog_name.schema_name"), "table_name") You can also use [`DBI::Id()`](https://dbi.r-dbi.org/reference/Id.html) , whose syntax has also evolved over time: con |> tbl(DBI::Id(database = "catalog_name", schema = "schema_name", table = "table_name")) con |> tbl(DBI::Id(catalog = "catalog_name", schema = "schema_name", table = "table_name")) con |> tbl(DBI::Id("catalog_name", "schema_name", "table_name")) Many of these options were poorly supported (i.e. we would accidentally break them from time-to-time) and suffered from the lack of a holistic vision. This release aims to bring order to the chaos by providing a succinct new syntax for literal table identifiers: [`I()`](https://rdrr.io/r/base/AsIs.html) . This allows you to succinctly identify a table nested inside a schema or catalog: con |> tbl(I("catalog_name.schema_name.table_name")) con |> tbl(I("schema_name.table_name")) [`I()`](https://rdrr.io/r/base/AsIs.html) is a base function, and you may be familiar with it from modelling, e.g. `lm(y ~ x + I(y * z))`. It performs a similar role for both dbplyr and modelling function: it tells the function to treat the argument as is, rather than quoting it in the case of dbplyr, or interpreting as an interaction in the case of [`lm()`](https://rdrr.io/r/stats/lm.html) . [`I()`](https://rdrr.io/r/base/AsIs.html) is dbplyr’s preferred way of specifying nested table identifiers and we will eventually formally supersede and then one day deprecate the other options. However, because their usage is widespread, this process will be slow and gradual, and play out over multiple years; there’s no need to make changes now. (If you’re the author of a dbplyr backend, you’ll can take advantage of this new syntax by using the `dbplyr_table_path` class. dbplyr now provides a [few helper functions](https://dbplyr.tidyverse.org/reference/is_table_path.html) to make this easier.) Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 46 folks who helped to make this release possible with their thoughtful comments and code contributions! [@aarizvi](https://github.com/aarizvi) , [@abalter](https://github.com/abalter) , [@andreassoteriadesmoj](https://github.com/andreassoteriadesmoj) , [@andrew-schulman](https://github.com/andrew-schulman) , [@apalacio9502](https://github.com/apalacio9502) , [@carlinstarrs](https://github.com/carlinstarrs) , [@catalamarti](https://github.com/catalamarti) , [@chicotobi](https://github.com/chicotobi) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dmenne](https://github.com/dmenne) , [@edgararuiz](https://github.com/edgararuiz) , [@edonnachie](https://github.com/edonnachie) , [@eitsupi](https://github.com/eitsupi) , [@ejneer](https://github.com/ejneer) , [@erydit](https://github.com/erydit) , [@espinielli](https://github.com/espinielli) , [@fh-afrachioni](https://github.com/fh-afrachioni) , [@ghost](https://github.com/ghost) , [@godislobster](https://github.com/godislobster) , [@gorcha](https://github.com/gorcha) , [@hadley](https://github.com/hadley) , [@hild0146](https://github.com/hild0146) , [@JakeHurlbut](https://github.com/JakeHurlbut) , [@jarodmeng](https://github.com/jarodmeng) , [@Jiefei-Wang](https://github.com/Jiefei-Wang) , [@joshbal](https://github.com/joshbal) , [@kelseyroberts](https://github.com/kelseyroberts) , [@kmishra9](https://github.com/kmishra9) , [@krlmlr](https://github.com/krlmlr) , [@m-muecke](https://github.com/m-muecke) , [@maciekbanas](https://github.com/maciekbanas) , [@marcusmunch](https://github.com/marcusmunch) , [@mgarbuzov](https://github.com/mgarbuzov) , [@mgirlich](https://github.com/mgirlich) , [@misea](https://github.com/misea) , [@MKatz-DHSC](https://github.com/MKatz-DHSC) , [@Mkranj](https://github.com/Mkranj) , [@multimeric](https://github.com/multimeric) , [@nathanhaigh](https://github.com/nathanhaigh) , [@nilescbn](https://github.com/nilescbn) , [@talegari](https://github.com/talegari) , [@Tazinho](https://github.com/Tazinho) , [@thomashulst](https://github.com/thomashulst) , [@Thranholm](https://github.com/Thranholm) , [@tomshafer](https://github.com/tomshafer) , and [@wstvcg](https://github.com/wstvcg) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyverse-dev-day [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyverse-Dev-Day [![](/blog/2024/04/tdd-2024/thumbnail-sq.jpg)](/blog/2024/04/tdd-2024/) [Tidyverse developer day 2024](/blog/2024/04/tdd-2024/) [package](/categories/package) Hadley Wickham Join the tidyverse team in Seattle on August 15 for a day of fun and programming to improve the tidyverse! [Read more ...](/blog/2024/04/tdd-2024/) 2024/04/09 [![](/blog/2020/02/tidy-dev-days-2020/thumbnail-sq.jpg)](/blog/2020/02/tidy-dev-days-2020/) [More 2020 tidy dev days!](/blog/2020/02/tidy-dev-days-2020/) [other](/categories/other) Mara Averick A roundup of our post-rstudio::conf tidy dev day, and announcement of a pre-useR! 2020 event. [Read more ...](/blog/2020/02/tidy-dev-days-2020/) 2020/02/28 [![](/blog/2019/11/tidyverse-dev-day-2020/thumb-sq.jpg)](/blog/2019/11/tidyverse-dev-day-2020/) [Tidyverse dev day 2020](/blog/2019/11/tidyverse-dev-day-2020/) [other](/categories/other) Mara Averick Join us for tidyverse developer day in San Francisco on January 31, 2020! [Read more ...](/blog/2019/11/tidyverse-dev-day-2020/) 2019/11/05 [![](/blog/2019/09/tidy-dev-day-toulouse/tidy-dev-day-toulouse-sq.jpg)](/blog/2019/09/tidy-dev-day-toulouse/) [Tidy dev day take two: Toulouse](/blog/2019/09/tidy-dev-day-toulouse/) [other](/categories/other) Mara Averick and Hadley Wickham Reflections on tidyverse dev day at useR! 2019. [Read more ...](/blog/2019/09/tidy-dev-day-toulouse/) 2019/09/06 [![](/blog/2019/04/tidyverse-dev-day-at-user-2019/tidyverse-dev-day-at-user-2019-sq.jpg)](/blog/2019/04/tidyverse-dev-day-at-user-2019/) [Tidyverse dev day at useR! 2019](/blog/2019/04/tidyverse-dev-day-at-user-2019/) [other](/categories/other) Mara Averick Join us for a pre-useR! tidyverse developer day on July 8! [Read more ...](/blog/2019/04/tidyverse-dev-day-at-user-2019/) 2019/04/24 [![](/blog/2018/11/tidyverse-developer-day-2019/tidyverse-developer-day-2019-sq.jpg)](/blog/2018/11/tidyverse-developer-day-2019/) [Tidyverse developer day 2019](/blog/2018/11/tidyverse-developer-day-2019/) [other](/categories/other) Mara Averick Join us for tidyverse developer day on January 19! [Read more ...](/blog/2018/11/tidyverse-developer-day-2019/) 2018/11/28 [![](/blog/2018/08/tidyverse-developer-day/tidyverse-developer-day-sq.jpg)](/blog/2018/08/tidyverse-developer-day/) [Save the date: tidyverse developer day](/blog/2018/08/tidyverse-developer-day/) [other](/categories/other) Mara Averick Save the date for tidyverse developer day the Saturday following rstudio::conf! [Read more ...](/blog/2018/08/tidyverse-developer-day/) 2018/08/27 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # nanoparquet 0.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) nanoparquet 0.3.0 ================= ![](/blog/2024/06/nanoparquet-0-3-0/thumbnail-wd.jpg) Photo by [Marina Zvada](https://www.pexels.com/photo/clock-between-columns-20134435/)   2024/06/20   [parquet](/tags/parquet/)   Gábor Csárdi We’re extremely pleased to announce the release of [nanoparquet](https://r-lib.github.io/nanoparquet/) 0.3.0. nanoparquet is a new R package that reads Parquet files into data frames, and writes data frames to Parquet files. You can install it from CRAN with: install.packages("nanoparquet") This blog post will cover the features and limitations of nanoparquet, and also our future plans. library(nanoparquet) What is Parquet?[](#what-is-parquet) ------------------------------------- Parquet is a file format for storing data on disk. It is specifically designed for large data sets, read-heavy workloads and data analysis. The most important features of Parquet are: * **Columnar**. Data is stored column-wise, so whole columns (or large chunks of columns) are easy to read quickly. Columnar storage allows better compression, fast operations on a subset of columns, and easy ways of removing columns or adding new columns to a data file. * **Binary**. A Parquet file is not a text file. Each Parquet data type is stored in a well-defined binary storage format, leaving no ambiguity about how fields are persed. * **Rich types**. Parquet supports a small set of _low level_ data types with well specified storage formats and encodings. On top of the low level types, it implemented several higher level logical types, like UTF-8 strings, time stamps, JSON strings, ENUM types (factors), etc. * **Well supported**. At this point Parquet is well supported across modern languages like R, Python, Rust, Java, Go, etc. In particular, Apache Arrow handles Parquet files very well, and has bindings to many languages. DuckDB is a very portable tool that reads and writes Parquet files, or even opens a set of Parquet files as a database. * **Performant**. Parquet columns may use various encodings and compression to ensure that the data files are kept as small as possible. When running an analytical query on the subset of the data, the Parquet format makes it easy to skip the columns and/or rows that are irrelevant. * **Concurrency built in**. A Parquet file can be divided into row groups. Parquet readers can read, uncompress and decode row groups in parallel. Parquet writes can encode and compress row groups in parallel. Even a single column may be divided into multiple pages, that can be (un)compressed, encode and decode in parallel. * **Missing values**. Support for missing values is built into the Parquet format. Why we created nanoparquet?[](#why-we-created-nanoparquet) ----------------------------------------------------------- Although Parquet is well supported by modern languages, today the complexity of the Parquet format often outweighs its benefits for smaller data sets. Many tools that support Parquet are typically used for larger, out of memory data sets, so there is a perception that Parquet is only for big data. These tools typically take longer to compile or install, and often seem too heavy for in-memory data analysis. With nanoparquet, we wanted to have a smaller tool that has no dependencies and is easy to install. Our goal is to facilitate the adoption of Parquet for smaller data sets, especially for teams that share data between multiple environments, e.g. R, Python, Java, etc. nanoparquet Features[](#nanoparquet-features) ---------------------------------------------- These are some of the nanoparquet features that we are most excited about. * **Lightweight**. nanoparquet has no package or system dependencies other than a C++-11 compiler. It compiles in about 30 seconds into an R package that is less than a megabyte in size. * **Reads many Parquet files**. [`nanoparquet::read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) supports reading most Parquet files. In particular, in supports all Parquet encodings and at the time of writing it supports three compression codecs: Snappy, Gzip and Zstd. Make sure you read “Limitations” below. * **Writes many R data types**. [`nanoparquet::write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) supports writing most R data frames. In particular, missing values are handled properly, factor columns are kept as factors, and temporal types are encoded correctly. Make sure you read “Limitations” below. * **Type mappings**. nanoparquet has a well defined set of [type mapping rules](https://r-lib.github.io/nanoparquet/reference/nanoparquet-types.html) . Use the [`parquet_column_types()`](https://r-lib.github.io/nanoparquet/dev/reference/parquet_column_types.html) function to see how [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) and [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) maps Parquet and R types for a file or a data frame. * **Metadata queries**. nanoparquet has a [number of functions](https://r-lib.github.io/nanoparquet/dev/reference/index.html#extract-parquet-metadata) that allow you to query the metadata and schema without reading in the full dataset. Examples[](#examples) ---------------------- ### Reading a Parquet file[](#reading-a-parquet-file) The nanoparquet R package contains an example Parquet file. We are going to use it to demonstrate how the package works. If the pillar package is loaded, then nanoparquet data frames are pretty-printed. library(nanoparquet) library(pillar) udf <- system.file("extdata/userdata1.parquet", package = "nanoparquet") Before actually reading the file, let’s look up some metadata about it, and also how its columns will be mapped to R types: parquet_info(udf) #> # A data frame: 1 × 7 #> file_name num_cols num_rows num_row_groups file_size parquet_version #> #> 1 /Users/gaborcsardi… 13 1000 1 73217 1 #> # ℹ 1 more variable: created_by parquet_column_types(udf) #> # A data frame: 13 × 6 #> file_name name type r_type repetition_type logical_type #> * > #> 1 /Users/gaborcsa… regi… INT64 POSIX… REQUIRED #> 2 /Users/gaborcsa… id INT32 integ… REQUIRED #> 3 /Users/gaborcsa… firs… BYTE… chara… OPTIONAL #> 4 /Users/gaborcsa… last… BYTE… chara… REQUIRED #> 5 /Users/gaborcsa… email BYTE… chara… OPTIONAL #> 6 /Users/gaborcsa… gend… BYTE… factor OPTIONAL #> 7 /Users/gaborcsa… ip_a… BYTE… chara… REQUIRED #> 8 /Users/gaborcsa… cc BYTE… chara… OPTIONAL #> 9 /Users/gaborcsa… coun… BYTE… chara… REQUIRED #> 10 /Users/gaborcsa… birt… INT32 Date OPTIONAL #> 11 /Users/gaborcsa… sala… DOUB… double OPTIONAL #> 12 /Users/gaborcsa… title BYTE… chara… OPTIONAL #> 13 /Users/gaborcsa… comm… BYTE… chara… OPTIONAL For every Parquet column we see its low level Parquet data type in `type`, e.g. `INT64` or `BYTE_ARRAY`. `r_type` the R type that [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) will create for that column. If `repetition_type` is `REQUIRED`, then that column cannot contain missing values. `OPTIONAL` columns may have missing values. `logical_type` is the higher level Parquet data type. E.g. the first column is an UTC (because of the `TRUE`) timestamp, in microseconds. It is stored as a 64 bit integer in the file, and it will be converted to a `POSIXct` object by [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) . To actually read the file into a data frame, call [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) : ud1 <- read_parquet(udf) ud1 #> # A data frame: 1,000 × 13 #> registration id first_name last_name email gender ip_address cc #> #> 1 2016-02-03 07:55:29 1 Amanda Jordan ajord… Female 1.197.201… 6759… #> 2 2016-02-03 17:04:03 2 Albert Freeman afree… Male 218.111.1… NA #> 3 2016-02-03 01:09:31 3 Evelyn Morgan emorg… Female 7.161.136… 6767… #> 4 2016-02-03 00:36:21 4 Denise Riley drile… Female 140.35.10… 3576… #> 5 2016-02-03 05:05:31 5 Carlos Burns cburn… NA 169.113.2… 5602… #> 6 2016-02-03 07:22:34 6 Kathryn White kwhit… Female 195.131.8… 3583… #> 7 2016-02-03 08:33:08 7 Samuel Holmes sholm… Male 232.234.8… 3582… #> 8 2016-02-03 06:47:06 8 Harry Howell hhowe… Male 91.235.51… NA #> 9 2016-02-03 03:52:53 9 Jose Foster jfost… Male 132.31.53… NA #> 10 2016-02-03 18:29:47 10 Emily Stewart estew… Female 143.28.25… 3574… #> # ℹ 990 more rows #> # ℹ 5 more variables: country , birthdate , salary , #> # title , comments ### Writing a Parquet file[](#writing-a-parquet-file) To show [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) , we’ll use the `flights` data in the nycflights13 package: library(nycflights13) flights #> # A tibble: 336,776 × 19 #> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time #> #> 1 2013 1 1 517 515 2 830 819 #> 2 2013 1 1 533 529 4 850 830 #> 3 2013 1 1 542 540 2 923 850 #> 4 2013 1 1 544 545 -1 1004 1022 #> 5 2013 1 1 554 600 -6 812 837 #> 6 2013 1 1 554 558 -4 740 728 #> 7 2013 1 1 555 600 -5 913 854 #> 8 2013 1 1 557 600 -3 709 723 #> 9 2013 1 1 557 600 -3 838 846 #> 10 2013 1 1 558 600 -2 753 745 #> # ℹ 336,766 more rows #> # ℹ 11 more variables: arr_delay , carrier , flight , #> # tailnum , origin , dest , air_time , distance , #> # hour , minute , time_hour First we check how columns of `flights` will be mapped to Parquet types: parquet_column_types(flights) #> # A data frame: 19 × 6 #> file_name name type r_type repetition_type logical_type #> > #> 1 NA year INT32 integ… REQUIRED #> 2 NA month INT32 integ… REQUIRED #> 3 NA day INT32 integ… REQUIRED #> 4 NA dep_time INT32 integ… OPTIONAL #> 5 NA sched_dep_t… INT32 integ… REQUIRED #> 6 NA dep_delay DOUB… double OPTIONAL #> 7 NA arr_time INT32 integ… OPTIONAL #> 8 NA sched_arr_t… INT32 integ… REQUIRED #> 9 NA arr_delay DOUB… double OPTIONAL #> 10 NA carrier BYTE… chara… REQUIRED #> 11 NA flight INT32 integ… REQUIRED #> 12 NA tailnum BYTE… chara… OPTIONAL #> 13 NA origin BYTE… chara… REQUIRED #> 14 NA dest BYTE… chara… REQUIRED #> 15 NA air_time DOUB… double OPTIONAL #> 16 NA distance DOUB… double REQUIRED #> 17 NA hour DOUB… double REQUIRED #> 18 NA minute DOUB… double REQUIRED #> 19 NA time_hour INT64 POSIX… REQUIRED This looks fine, so we go ahead and write out the file. By default it will be Snappy-compressed, and many columns will be dictionary encoded. write_parquet(flights, "flights.parquet") ### Parquet metadata[](#parquet-metadata) Use [`parquet_schema()`](https://r-lib.github.io/nanoparquet/reference/parquet_schema.html) to see the schema of a Parquet file. The schema also includes “internal” parquet columns. Every Parquet file is a tree where columns may be part of an “internal” column. nanoparquet currently only supports flat files, that consist of a single internal root column and all other columns are leaf columns and are children of the root: parquet_schema("flights.parquet") #> # A data frame: 20 × 11 #> file_name name type type_length repetition_type converted_type #> #> 1 flights.parquet schema NA NA NA NA #> 2 flights.parquet year INT32 NA REQUIRED INT_32 #> 3 flights.parquet month INT32 NA REQUIRED INT_32 #> 4 flights.parquet day INT32 NA REQUIRED INT_32 #> 5 flights.parquet dep_time INT32 NA OPTIONAL INT_32 #> 6 flights.parquet sched_dep_t… INT32 NA REQUIRED INT_32 #> 7 flights.parquet dep_delay DOUB… NA OPTIONAL NA #> 8 flights.parquet arr_time INT32 NA OPTIONAL INT_32 #> 9 flights.parquet sched_arr_t… INT32 NA REQUIRED INT_32 #> 10 flights.parquet arr_delay DOUB… NA OPTIONAL NA #> 11 flights.parquet carrier BYTE… NA REQUIRED UTF8 #> 12 flights.parquet flight INT32 NA REQUIRED INT_32 #> 13 flights.parquet tailnum BYTE… NA OPTIONAL UTF8 #> 14 flights.parquet origin BYTE… NA REQUIRED UTF8 #> 15 flights.parquet dest BYTE… NA REQUIRED UTF8 #> 16 flights.parquet air_time DOUB… NA OPTIONAL NA #> 17 flights.parquet distance DOUB… NA REQUIRED NA #> 18 flights.parquet hour DOUB… NA REQUIRED NA #> 19 flights.parquet minute DOUB… NA REQUIRED NA #> 20 flights.parquet time_hour INT64 NA REQUIRED TIMESTAMP_MIC… #> # ℹ 5 more variables: logical_type >, num_children , scale , #> # precision , field_id To see more information about a Parquet file, use [`parquet_metadata()`](https://r-lib.github.io/nanoparquet/reference/parquet_metadata.html) : parquet_metadata("flights.parquet") #> $file_meta_data #> # A data frame: 1 × 5 #> file_name version num_rows key_value_metadata created_by #> > #> 1 flights.parquet 1 336776 https://github.com/gaborc… #> #> $schema #> # A data frame: 20 × 11 #> file_name name type type_length repetition_type converted_type #> #> 1 flights.parquet schema NA NA NA NA #> 2 flights.parquet year INT32 NA REQUIRED INT_32 #> 3 flights.parquet month INT32 NA REQUIRED INT_32 #> 4 flights.parquet day INT32 NA REQUIRED INT_32 #> 5 flights.parquet dep_time INT32 NA OPTIONAL INT_32 #> 6 flights.parquet sched_dep_t… INT32 NA REQUIRED INT_32 #> 7 flights.parquet dep_delay DOUB… NA OPTIONAL NA #> 8 flights.parquet arr_time INT32 NA OPTIONAL INT_32 #> 9 flights.parquet sched_arr_t… INT32 NA REQUIRED INT_32 #> 10 flights.parquet arr_delay DOUB… NA OPTIONAL NA #> 11 flights.parquet carrier BYTE… NA REQUIRED UTF8 #> 12 flights.parquet flight INT32 NA REQUIRED INT_32 #> 13 flights.parquet tailnum BYTE… NA OPTIONAL UTF8 #> 14 flights.parquet origin BYTE… NA REQUIRED UTF8 #> 15 flights.parquet dest BYTE… NA REQUIRED UTF8 #> 16 flights.parquet air_time DOUB… NA OPTIONAL NA #> 17 flights.parquet distance DOUB… NA REQUIRED NA #> 18 flights.parquet hour DOUB… NA REQUIRED NA #> 19 flights.parquet minute DOUB… NA REQUIRED NA #> 20 flights.parquet time_hour INT64 NA REQUIRED TIMESTAMP_MIC… #> # ℹ 5 more variables: logical_type >, num_children , scale , #> # precision , field_id #> #> $row_groups #> # A data frame: 1 × 7 #> file_name id total_byte_size num_rows file_offset total_compressed_size #> #> 1 flights.parq… 0 5732430 336776 NA NA #> # ℹ 1 more variable: ordinal #> #> $column_chunks #> # A data frame: 19 × 19 #> file_name row_group column file_path file_offset offset_index_offset #> #> 1 flights.parquet 0 0 NA 23 NA #> 2 flights.parquet 0 1 NA 111 NA #> 3 flights.parquet 0 2 NA 323 NA #> 4 flights.parquet 0 3 NA 6738 NA #> 5 flights.parquet 0 4 NA 468008 NA #> 6 flights.parquet 0 5 NA 893557 NA #> 7 flights.parquet 0 6 NA 1312660 NA #> 8 flights.parquet 0 7 NA 1771896 NA #> 9 flights.parquet 0 8 NA 2237931 NA #> 10 flights.parquet 0 9 NA 2653250 NA #> 11 flights.parquet 0 10 NA 2847249 NA #> 12 flights.parquet 0 11 NA 3374563 NA #> 13 flights.parquet 0 12 NA 3877832 NA #> 14 flights.parquet 0 13 NA 3966418 NA #> 15 flights.parquet 0 14 NA 4264662 NA #> 16 flights.parquet 0 15 NA 4639410 NA #> 17 flights.parquet 0 16 NA 4976781 NA #> 18 flights.parquet 0 17 NA 5120936 NA #> 19 flights.parquet 0 18 NA 5427022 NA #> # ℹ 13 more variables: offset_index_length , column_index_offset , #> # column_index_length , type , encodings >, #> # path_in_schema >, codec , num_values , #> # total_uncompressed_size , total_compressed_size , #> # data_page_offset , index_page_offset , #> # dictionary_page_offset The output will include the schema, as above, but also data about the row groups ( [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) always writes a single row group currently), and column chunks. There is one column chunk per column in each row group. The columns chunk information also tells you whether a column chunk is dictionary encoded, its encoding, its size, etc. cc <- parquet_metadata("flights.parquet")$column_chunks cc[, c("column", "encodings", "dictionary_page_offset")] #> # A data frame: 19 × 3 #> column encodings dictionary_page_offset #> > #> 1 0 4 #> 2 1 48 #> 3 2 181 #> 4 3 1445 #> 5 4 463903 #> 6 5 891412 #> 7 6 1306995 #> 8 7 1767223 #> 9 8 2235594 #> 10 9 2653154 #> 11 10 2831850 #> 12 11 3352496 #> 13 12 3877796 #> 14 13 3965856 #> 15 14 4262597 #> 16 15 4638461 #> 17 16 4976675 #> 18 17 5120660 #> 19 18 5379476 cc[["encodings"]][1:3] #> [[1]] #> [1] "PLAIN" "RLE" "RLE_DICTIONARY" #> #> [[2]] #> [1] "PLAIN" "RLE" "RLE_DICTIONARY" #> #> [[3]] #> [1] "PLAIN" "RLE" "RLE_DICTIONARY" Limitations[](#limitations) ---------------------------- nanoparquet 0.3.0 has a number of limitations. * **Only flat tables**. [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) can only read flat tables, i.e. Parquet files without nested columns. (Technically all Parquet files are nested, and nanoparquet supports exactly one level of nesting: a single meta column that contains all other columns.) Similarly, [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) will not write list columns. * **Unsupported Parquet types**. [`read_parquet()`](https://r-lib.github.io/nanoparquet/reference/read_parquet.html) reads some Parquet types as raw vectors of a list column currently, e.g. `FLOAT16`, `INTERVAL`. See [the manual](https://r-lib.github.io/nanoparquet/reference/nanoparquet-types.html) for details. * **No encryption**. Encrypted Parquet files are not supported. * **Missing compression codecs**. `LZO`, `BROTLI` and `LZ4` compression is not yet supported. * **No statistics**. nanoparquet does not read or write statistics, e.g. minimum and maximum values from and to Parquet files. * **No checksums**. nanoparquet does not check or write checksums currently. * **No Bloom filters**. nanoparquet does not currently support reading or writing Bloom filters from or to Parquet files. * **May be slow for large files**. Being single-threaded and not fully optimized, nanoparquet is probably not suited well for large data sets. It should be fine for a couple of gigabytes. It may be fine if all the data fits into memory comfortably. * **Single row group**. [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) always creates a single row group, which is not optimal for large files. * **Automatic encoding**. It is currently not possible to choose encodings in [`write_parquet()`](https://r-lib.github.io/nanoparquet/reference/write_parquet.html) manually. We are planning on solving these limitations, while keeping nanoparquet as lean as possible. In particular, if you find a Parquet file that nanoparquet cannot read, please report an issue in our [issue tracker](https://github.com/r-lib/nanoparquet/issues) ! Other tools for Parquet files[](#other-tools-for-parquet-files) ---------------------------------------------------------------- If you run into some of these limitations, chances are you are dealing with a larget data set, and you will probably benefit from using tools geared towards larger Parquet files. Luckily you have several options. ### In R[](#in-r) #### Apache Arrow[](#apache-arrow) You can usually install the `arrow` package from CRAN. Note, however, that some CRAN builds are suboptimal at the time of writing, e.g. the macOS builds lack Parquet support and it is best to install arrow from [R-universe](https://apache.r-universe.dev/arrow) on these platforms. Call [`arrow::read_parquet()`](https://arrow.apache.org/docs/r/reference/read_parquet.html) to read Parquet files, and [`arrow::write_parquet()`](https://arrow.apache.org/docs/r/reference/write_parquet.html) to write them. You can also use [`arrow::open_dataset()`](https://arrow.apache.org/docs/r/reference/open_dataset.html) to open (one or more) Parquet files and perform queries on them without loading all data into memory. #### DuckDB[](#duckdb) DuckDB is an excellent tool that handles Parquet files seemlessly. You can install the duckdb R package from CRAN. To read a Parquet file into an R data frame with DuckDB, call df <- duckdb:::sql("FROM 'file.parquet'") Alternatively, you can open (one or more) Parquet files and query them as a DuckDB database, potentially without reading all data into memory at once. Here is an example that shows how to put an R data frame into a (temporary) DuckDB database, and how to export it to Parquet: drv <- duckdb::duckdb() con <- DBI::dbConnect(drv) on.exit(DBI::dbDisconnect(con), add = TRUE) DBI::dbWriteTable(con, "mtcars", mtcars) DBI::dbExecute(con, DBI::sqlInterpolate(con, "COPY mtcars TO ?filename (FORMAT 'parquet', COMPRESSION 'snappy')", filename = 'mtcars.parquet' )) ### In Python[](#in-python) There are at least three good options to handle Parquet files in Python. Just like for R, the first two are [Apache Arrow](https://arrow.apache.org/docs/python/index.html) and [DuckDB](https://duckdb.org/docs/api/python/overview.html) . You can also try the [fastparquet](https://pypi.org/project/fastparquet/) Python package for a potentially lighter solution. Acknowledgements[](#acknowledgements) -------------------------------------- nanoparquet would not exist without the work of Hannes Mühleisen on [miniparquet](https://github.com/hannes/miniparquet) , which had similar goals, but it is discontinued now. nanoparquet is a fork of miniparquet. nanoparquet also contains code and test Parquet files from DuckDB, Apache Parquet, Apache Arrow, Apache Thrift, it contains libraries from Google, Facebook, etc. see the [COPYRIGHTS file](https://github.com/r-lib/nanoparquet/blob/main/inst/COPYRIGHTS) in the repository for the full details. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Survival analysis for time-to-event data with tidymodels [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Survival analysis for time-to-event data with tidymodels ======================================================== ![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-wd.jpg) Photo by [Heather Zabriskie](https://unsplash.com/photos/vintage-brown-and-white-watch-lot-yBzrPGLjMQw)   2024/04/03   [tidymodels](/tags/tidymodels/) , [workflows](/tags/workflows/) , [parsnip](/tags/parsnip/) , [censored](/tags/censored/) , [workflowsets](/tags/workflowsets/) , [tune](/tags/tune/) , [yardstick](/tags/yardstick/)   Hannah Frick We’re tickled pink to announce the support of survival analysis for time-to-event data across tidymodels. The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. This new support makes survival analysis a first-class citizen in tidymodels and gives censored regression modeling the same flexibility and ease as classification or regression. The functionality resides in multiple tidymodels packages. The easiest way to install them all is to install the tidymodels meta-package: install.packages("tidymodels") This blog post will highlight why this is useful, explain which additions we’ve made to the framework, and point to several places to learn more. You can see a full list of changes in the release notes: * [parsnip](https://parsnip.tidymodels.org/news/index.html#parsnip-120) * [censored](https://censored.tidymodels.org/news/index.html#censored-030) * [yardstick](https://yardstick.tidymodels.org/news/index.html#yardstick-130) * [workflows](https://workflows.tidymodels.org/news/index.html#workflows-114) * [tune](https://tune.tidymodels.org/news/index.html#tune-120) * [finetune](https://finetune.tidymodels.org/news/index.html#finetune-120) * [workflowsets](https://workflowsets.tidymodels.org/news/index.html#workflowsets-110) Increasing usefulness: Two perspectives[](#increasing-usefulness-two-perspectives) ----------------------------------------------------------------------------------- We’d like to situate the changes from two different perspectives: How this is useful for people already familiar with survival analysis as well as for people already familiar with tidymodels. If you are already familiar with both: Excellent, this is very much for you! Read on for more details on how these two things come together. ### Adding tidymodels to your tool kit[](#adding-tidymodels-to-your-tool-kit) If you are already familiar with survival analysis but maybe not tidymodels, these changes now unlock a whole framework for predictive modelling for you. It applies tidyverse principles to modeling, meaning it strives to be consistent, composable, and human-centered. The framework covers the modeling process from the initial test/train split of the data all the way to tuning various models. Along the way it offers a rich selection of preprocessing techniques, resampling schemes, and performance metrics along with safe-guards against accidental overfitting. We make the full case for tidymodels at [tidymodels.org](https://www.tidymodels.org/) . ### Adding survival analysis to your tool kit[](#adding-survival-analysis-to-your-tool-kit) If you are already familiar with tidymodels but maybe not survival analysis, these changes let you leverage the familiar framework for an additional type of modeling problem. Survival analysis offers methods for modeling time-to-event data. While it has its roots in medical research, it has broad applications as that event of interest can be so much more than a medical outcome. Take customer churn as an example: We are interested in how long someone is a customer for and when they churn. For customers who churned, we have the complete time for which they were customers. For existing customers, we only know how long they’ve been customers for _so far_. Such observations are called censored. So what are our modeling choices here? We could look at the time and model that as a regression problem. We could look at the event status and model that as a classification problem. Both options might get us somewhere close to an answer to our original modeling question but not quite there. Censored regression models let us model an outcome that includes both aspects, the time and the event status. And with that, it can deal with both censored and uncensored observations appropriately. With this type of model, we can predict the survival time, or in more applied terms, how long someone will stay as a customer. We can also predict the probability of survival at a given time point. This lets us answer questions like “How likely is it that this customer will churn after 3 months?". See which prediction types are available for which models at [censored.tidymodels.org](https://censored.tidymodels.org/) . Ch-ch-changes: What’s new for censored regression?[](#ch-ch-changes-whats-new-for-censored-regression) ------------------------------------------------------------------------------------------------------- The main components needed for this full-fledged integration of survival analysis into tidymodels were * Survival analysis models that can take censoring into account * Survival analysis performance metrics that can take censoring into account * Integrating changes required by these models and metrics into the framework For the models, parsnip gained a new mode, `"censored regression"`, for existing models as well as new model types such as `proportional_hazards()`. Engines for these reside in censored, the parsnip extension package for survival models. The `"censored regression"` mode has been around for a while and we’ve previously shared posts on [our initial thoughts](https://www.tidyverse.org/blog/2021/11/survival-analysis-parsnip-adjacent/) and the [release of censored](https://www.tidyverse.org/blog/2022/08/censored-0-1-0/) . Now we’ve added the metrics: [yardstick v1.3.0](https://yardstick.tidymodels.org/news/index.html#yardstick-130) includes new metrics for assessing censored regression models. Somewhat similar to how metrics for classification models can take class predictions or probability predictions as input, these survival metrics can take predicted survival times or predictions of survival probabilities as input. The new metrics are * Concordance index on the survival time via `concordance_survival()` * Brier score on the survival probability and its integrated version via `brier_survival()` and `brier_survival_integrated()` * ROC curve and the area under the ROC curve on the survival probabilities via `roc_curve_survival()` and `auc_roc_survival()` respectively The probability of survival is always defined _at a certain point in time_. We call that time point the _evaluation time_ because it is then also the time point at which we want to evaluate model performance. Metrics that work on the survival probabilities are also called _dynamic metrics_ and you can read more about them here: * [Dynamic Performance Metrics for Event Time Data](https://www.tidymodels.org/learn/statistics/survival-metrics/) * [Accounting for Censoring in Performance Metrics for Event Time Data](https://www.tidymodels.org/learn/statistics/survival-metrics-details/) The evaluation time is also the best example to illustrate the changes necessary to the framework. Most of them were under the hood but the evaluation time is user-facing. Let’s take a look at that. While the need for evaluation times is dependent on type of metric, it is not actually specified as an argument to the metric functions. Like yardstick’s other metrics, those take pre-made predictions as the input. So where do you specify it then? * You need to specify it to directly predict survival probabilities, via [`predict()`](https://rdrr.io/r/stats/predict.html) or `augment()`. We introduced the corresponding `eval_time` argument first for fitted models in [parsnip and censored](https://www.tidyverse.org/blog/2023/04/censored-0-2-0/#introducing-eval_time) and have added it now for workflows. * You also need to specify it for the tuning functions `tune_*()` from tune and finetune as they will predict survival probabilities as part of the tuning process. * Lastly, the `eval_time` argument now shows up when working with tuning/resampling results such as in `show_best()` or `autoplot()`. Those changes span the packages generating and working with resampling results: tune, finetune, and workflowsets. As we said, plenty of changes under the hood but you shouldn’t need to notice them. Everything else should work “as usual,” allowing the same ease and flexibility in combining tidymodels functionality for censored regression as for classification and regression. The pieces come together: A case study[](#the-pieces-come-together-a-case-study) --------------------------------------------------------------------------------- To see it all in action, check out the case study [How long until building complaints are dispositioned?](https://www.tidymodels.org/learn/statistics/survival-case-study/) on the tidymodels website! The city of New York publishes data on complaints received by the Department of Buildings that include how long it takes for a complaint to be dealt with (“dispositioned”) as well as several characteristics of the complaint. The case study covers a full analysis. We start with splitting the data into test and training sets, explore different preprocessing strategies and model types via tuning, and predict with a final model. It should give you a good first impression of how to use tidymodels for predictive survival analysis. We hope you’ll find this new capability of tidymodels useful! Acknowledgements[](#acknowledgements) -------------------------------------- Many thanks to the people who contributed to our packages since their last release: **parsnip:** [@AlbanOtt2](https://github.com/AlbanOtt2) , [@birbritto](https://github.com/birbritto) , [@christophscheuch](https://github.com/christophscheuch) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@Freestyleyang](https://github.com/Freestyleyang) , [@gmcmacran](https://github.com/gmcmacran) , [@hfrick](https://github.com/hfrick) , [@jmunyoon](https://github.com/jmunyoon) , [@joscani](https://github.com/joscani) , [@jxu](https://github.com/jxu) , [@marcelglueck](https://github.com/marcelglueck) , [@mattheaphy](https://github.com/mattheaphy) , [@mesdi](https://github.com/mesdi) , [@millermc38](https://github.com/millermc38) , [@nipnipj](https://github.com/nipnipj) , [@pgg1309](https://github.com/pgg1309) , [@rdavis120](https://github.com/rdavis120) , [@seb-mueller](https://github.com/seb-mueller) , [@SHo-JANG](https://github.com/SHo-JANG) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , [@vidarsumo](https://github.com/vidarsumo) , and [@wzbillings](https://github.com/wzbillings) . **censored:** [@bcjaeger](https://github.com/bcjaeger) , [@brunocarlin](https://github.com/brunocarlin) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@noahtsao](https://github.com/noahtsao) , and [@tripartio](https://github.com/tripartio) . **yardstick:** [@aecoleman](https://github.com/aecoleman) , [@asb2111](https://github.com/asb2111) , [@atsyplenkov](https://github.com/atsyplenkov) , [@bgreenwell](https://github.com/bgreenwell) , [@Dpananos](https://github.com/Dpananos) , [@EduMinsky](https://github.com/EduMinsky) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@heidekrueger](https://github.com/heidekrueger) , [@hfrick](https://github.com/hfrick) , [@iacrowe](https://github.com/iacrowe) , [@jarbet](https://github.com/jarbet) , [@jxu](https://github.com/jxu) , [@mattwarkentin](https://github.com/mattwarkentin) , [@maxwell-geospatial](https://github.com/maxwell-geospatial) , [@moloscripts](https://github.com/moloscripts) , [@rdavis120](https://github.com/rdavis120) , [@ruddnr](https://github.com/ruddnr) , [@SimonCoulombe](https://github.com/SimonCoulombe) , [@simonpcouch](https://github.com/simonpcouch) , [@tbrittoborges](https://github.com/tbrittoborges) , [@tonyelhabr](https://github.com/tonyelhabr) , [@tripartio](https://github.com/tripartio) , [@TSI-PTG](https://github.com/TSI-PTG) , [@vnijs](https://github.com/vnijs) , [@wbuchanan](https://github.com/wbuchanan) , and [@zkrog](https://github.com/zkrog) . **workflows:** [@Milardkh](https://github.com/Milardkh) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . **tune:** [@AlbertoImg](https://github.com/AlbertoImg) , [@dramanica](https://github.com/dramanica) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@epiheather](https://github.com/epiheather) , [@hfrick](https://github.com/hfrick) , [@joranE](https://github.com/joranE) , [@jrosell](https://github.com/jrosell) , [@jxu](https://github.com/jxu) , [@kbodwin](https://github.com/kbodwin) , [@kenraywilliams](https://github.com/kenraywilliams) , [@KJT-Habitat](https://github.com/KJT-Habitat) , [@lionel-](https://github.com/lionel-) , [@marcozanotti](https://github.com/marcozanotti) , [@MasterLuke84](https://github.com/MasterLuke84) , [@mikemahoney218](https://github.com/mikemahoney218) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , [@Peter4801](https://github.com/Peter4801) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@walkerjameschris](https://github.com/walkerjameschris) . **finetune:** [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@jdberson](https://github.com/jdberson) , [@jrosell](https://github.com/jrosell) , [@mfansler](https://github.com/mfansler) , [@ruddnr](https://github.com/ruddnr) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . **workflowsets:** [@dchiu911](https://github.com/dchiu911) , [@hfrick](https://github.com/hfrick) , [@jkylearmstrong](https://github.com/jkylearmstrong) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , and [@simonpcouch](https://github.com/simonpcouch) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # yardstick [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Yardstick [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2024/03/tidymodels-fairness/thumbnail-sq.jpg)](/blog/2024/03/tidymodels-fairness/) [Fair machine learning with tidymodels](/blog/2024/03/tidymodels-fairness/) [learn](/categories/learn) Simon Couch Recent tidymodels releases integrated a set of tools for assessing whether machine learning models treat groups of people differently. [Read more ...](/blog/2024/03/tidymodels-fairness/) 2024/03/21 [![](/blog/2023/04/tidymodels-2023-q1/thumbnail-sq.jpg)](/blog/2023/04/tidymodels-2023-q1/) [Q1 2023 tidymodels digest](/blog/2023/04/tidymodels-2023-q1/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/04/tidymodels-2023-q1/) 2023/04/28 [![](/blog/2021/07/tidymodels-2021-q2/thumbnail-sq.jpg)](/blog/2021/07/tidymodels-2021-q2/) [Q2 2021 tidymodels digest](/blog/2021/07/tidymodels-2021-q2/) [roundup](/categories/roundup) Julia Silge Releases of tidymodels packages in Q2 of 2021 include more streamlined memory footprints for feature-engineering recipes, new model engine options, and better support for post-processing predictions. [Read more ...](/blog/2021/07/tidymodels-2021-q2/) 2021/07/02 [![](/blog/2019/09/tidymodels-2019-09/tidymodels-2019-09-sq.jpg)](/blog/2019/09/tidymodels-2019-09/) [tidymodels updates](/blog/2019/09/tidymodels-2019-09/) [package](/categories/package) Max Kuhn, Edgar Ruiz, and Davis Vaughan The latest updates to the tidymodels packages [Read more ...](/blog/2019/09/tidymodels-2019-09/) 2019/09/05 [![](/blog/2018/11/tidymodels-update-nov-18/tidymodels-update-nov-18-sq.jpg)](/blog/2018/11/tidymodels-update-nov-18/) [tidymodels package updates](/blog/2018/11/tidymodels-update-nov-18/) [package](/categories/package) Max Kuhn, Davis Vaughan, Alex Hayes A variety of updates for tidymodels packages. [Read more ...](/blog/2018/11/tidymodels-update-nov-18/) 2018/11/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # learn [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Learn [![](/blog/2024/04/tidymodels-survival-analysis/thumbnail-sq.jpg)](/blog/2024/04/tidymodels-survival-analysis/) [Survival analysis for time-to-event data with tidymodels](/blog/2024/04/tidymodels-survival-analysis/) [learn](/categories/learn) Hannah Frick Recent releases integrate survival analysis into tidymodels. This now unlocks the framework for censored regression and provides modeling capabilities for time-to-event data. [Read more ...](/blog/2024/04/tidymodels-survival-analysis/) 2024/04/03 [![](/blog/2024/03/tidymodels-fairness/thumbnail-sq.jpg)](/blog/2024/03/tidymodels-fairness/) [Fair machine learning with tidymodels](/blog/2024/03/tidymodels-fairness/) [learn](/categories/learn) Simon Couch Recent tidymodels releases integrated a set of tools for assessing whether machine learning models treat groups of people differently. [Read more ...](/blog/2024/03/tidymodels-fairness/) 2024/03/21 [![](/blog/2023/08/teach-tidyverse-23/thumbnail-sq.jpg)](/blog/2023/08/teach-tidyverse-23/) [Teaching the tidyverse in 2023](/blog/2023/08/teach-tidyverse-23/) [learn](/categories/learn) Mine Çetinkaya-Rundel Recommendations for teaching the tidyverse in 2023, summarizing package updates most relevant for teaching data science with the tidyverse, particularly to new learners. [Read more ...](/blog/2023/08/teach-tidyverse-23/) 2023/08/07 [![](/blog/2023/08/data-trail/thumbnail-sq.jpg)](/blog/2023/08/data-trail/) [Solutions for R4DS, 2e with Data Trail](/blog/2023/08/data-trail/) [learn](/categories/learn) Jabir Ghaffar, Davon Person, Mine Çetinkaya-Rundel, Tracy Teal Jabir and Davon from Data Trail worked with us to create the solutions manual for R for Data Science, 2nd edition. This blog post summarizes their experience and shares their reflections from working on this project. [Read more ...](/blog/2023/08/data-trail/) 2023/08/02 [![](/blog/2023/07/r4ds-2e/thumbnail-sq.jpg)](/blog/2023/07/r4ds-2e/) [R for Data Science, 2nd edition](/blog/2023/07/r4ds-2e/) [learn](/categories/learn) Mine Çetinkaya-Rundel The second edition of R for Data Science is out, and it’s a major reworking of the first edition. [Read more ...](/blog/2023/07/r4ds-2e/) 2023/07/11 [![](/blog/2023/05/purrr-walk-this-way/thumbnail-sq.jpg)](/blog/2023/05/purrr-walk-this-way/) [`purrr::walk()` this way](/blog/2023/05/purrr-walk-this-way/) [learn](/categories/learn) Mara Averick How to use `purrr::walk()` to write many files, featuring file-system navigation with the fs package. [Read more ...](/blog/2023/05/purrr-walk-this-way/) 2023/05/26 [![](/blog/2023/03/cran-checks-compiled-code/thumbnail-sq.jpg)](/blog/2023/03/cran-checks-compiled-code/) [New CRAN requirements for packages with C and C++](/blog/2023/03/cran-checks-compiled-code/) [learn](/categories/learn) [programming](/categories/programming) Andy Teucher A few recent changes in CRAN requirements for packages containing C or C++ code have caused package developers some headaches. This post outlines the issues and provides solutions the tidyverse team has used to address them. [Read more ...](/blog/2023/03/cran-checks-compiled-code/) 2023/03/30 [![](/blog/2022/09/playing-on-the-same-team-as-your-dependecy/thumbnail-sq.jpg)](/blog/2022/09/playing-on-the-same-team-as-your-dependecy/) [Playing on the same team as your dependency](/blog/2022/09/playing-on-the-same-team-as-your-dependecy/) [learn](/categories/learn) Thomas Lin Pedersen Using another package as a dependency is a two-way street, but the expectations can be murky. This blog post guides you towards becoming a stellar reverse dependency. [Read more ...](/blog/2022/09/playing-on-the-same-team-as-your-dependecy/) 2022/09/29 [![](/blog/2022/09/its-about-time/thumbnail-sq.jpg)](/blog/2022/09/its-about-time/) [It’s about time](/blog/2022/09/its-about-time/) [learn](/categories/learn) Mara Averick Davis Vaughan’s talk from rstudio::conf(2022) on clock, an R package that aims to provide comprehensive and safe handling of date-times. [Read more ...](/blog/2022/09/its-about-time/) 2022/09/28 [![](/blog/2022/02/upkeep-testthat-3/thumbnail-sq.jpg)](/blog/2022/02/upkeep-testthat-3/) [Upgrading to testthat edition 3](/blog/2022/02/upkeep-testthat-3/) [learn](/categories/learn) Hannah Frick A workflow for upgrading to testthat edition 3: activation, deprecations, changes to warnings, messages, and comparisons. [Read more ...](/blog/2022/02/upkeep-testthat-3/) 2022/02/22 * [««](/categories/learn/) * « * [1](/categories/learn/) * [2](/categories/learn/page/2/) * [3](/categories/learn/page/3/) * [»](/categories/learn/page/2/) * [»»](/categories/learn/page/3/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Fair machine learning with tidymodels [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Fair machine learning with tidymodels ===================================== ![](/blog/2024/03/tidymodels-fairness/thumbnail-wd.jpg) Photo by [Patrick Fore](https://unsplash.com/photos/JBghIzjbuLs)   2024/03/21   [tidymodels](/tags/tidymodels/) , [tune](/tags/tune/) , [yardstick](/tags/yardstick/)   Simon Couch We’re very, very excited to announce the introduction of tools for assessing model fairness in tidymodels. This effort involved coordination from various groups at Posit over the course of over a year and resulted in a toolkit that we believe is both principled and impactful. Fairness assessment features for tidymodels extend across a number of packages; to install each, use the tidymodels meta-package: install.packages("tidymodels") Machine learning fairness[](#machine-learning-fairness) -------------------------------------------------------- In recent years, high-profile analyses have called attention to many contexts where the use of machine learning deepened inequities in our communities. In late 2022, a group of Posit employees across teams, roles, and technical backgrounds formed a reading group to engage with literature on machine learning fairness, a research field that aims to define what it means for a statistical model to act unfairly and take measures to address that unfairness. We then designed new software functionality and learning resources to help data scientists measure and critique the ways in which the machine learning models they’ve built might disparately impact people affected by that model. Perhaps the core question that fairness as a research field has tried to address is exactly what a machine learning model acting fairly entails. As a recent primer notes, “\[t\]he rapid growth of this new field has led to wildly inconsistent motivations, terminology, and notation, presenting a serious challenge for cataloging and comparing definitions” (Mitchell et al. 2021). Broadly, approaches to fairness provide tooling—whether social or algorithmic—to understand the social implications of utilizing a machine learning model. Different researchers categorize approaches to fairness differently, but work in this area can be loosely summarized as falling into one or more of the following categories: assessment, mitigation, and critique. * _Assessment_: Fairness assessment tooling allows practitioners to measure the degree to which a machine learning model acts unfairly given some definition of fairness. The chosen definition of fairness greatly impacts whether a model’s predictions are regarded as fair. While there have been many, many definitions of fairness proposed—a popular tutorial on these approaches compares 21 canonical definitions—most all of them involve simple inequalities based on a small set of conditional probabilities (Narayanan 2018; Mitchell et al. 2021). * _Mitigation_: Given a fairness assessment, mitigation approaches reduce the degree to which a machine learning model acts unfairly given some definition of fairness. Making a model more fair according to one metric may make that model less fair according to another. Approaches to mitigation are subject to impossibility theorems, which show that “definitions are not mathematically or morally compatible in general” (Mitchell et al. 2021). That is, there is no way to satisfy many fairness constraints at once unless we live in a world with no inequality to start with. However, more recent studies have shown that near-fairness with respect to several definitions is quite possible (Bell et al. 2023). * _Critique_: While approaches to assessment and mitigation seek to reduce complexity and situate notions of fairness in mathematical formalism, sociotechnical critique provides tooling to better understand how mathematical notions of fairness may fail to account for the real-world complexity of social phenomena. Work in this discipline often reveals that, in the process of measuring or addressing unfairness by some definition, methods for fairness assessment and mitigation may actually ignore, necessitate, or introduce unfairness by some other definition. The work of scoping Posit’s resources for fair machine learning, in large part, involved striking the right balance between tools in these categories and integrating them thoughtfully among our existing functionality. Rather than supporting as many fairness-oriented tools as possible, our goal is to best enable users of our tools to reason well about the fairness-relevant decisions they make throughout the modeling process. Additions to tidymodels[](#additions-to-tidymodels) ---------------------------------------------------- The most recent set of tidymodels releases include changes that provide support for assessment and critique using the tidymodels framework. The most recent yardstick release introduces [a tool to create fairness metrics](https://yardstick.tidymodels.org/reference/new_groupwise_metric.html) with the problem context in mind, as well as [some outputs of that tool](https://yardstick.tidymodels.org/reference/index.html#fairness-metrics) implementing common fairness metrics. For a higher-level introduction to the concept of a groupwise metric, we’ve also introduced a [new package vignette](https://yardstick.tidymodels.org/articles/grouping.html) . To see those fairness metrics in action, see [this new article on tidymodels.org](https://www.tidymodels.org/learn/work/fairness-detectors/) , a case study using data about GPT detectors. The most recent tune release integrates support for those fairness metrics from yardstick, allowing users to evaluate fairness criteria across resamples. To demonstrate those features in context, we’ve added [another new article on tidymodels.org](https://www.tidymodels.org/learn/work/fairness-readmission/) , modeling hospital readmission for patients with Type I diabetes. Notably, we haven’t introduced functionality to support mitigation. While a number of methods have proliferated over the years to finetune models to act more fairly with respect to some fairness criteria, each apply only in relatively niche applications with modest experimental results (Agarwal et al. 2018; Mittelstadt, Wachter, and Russell 2023). For now, we believe that, in practice, the efforts of practitioners—and thus our efforts to support them—are better spent engaging with the sociotechnical context of a given modeling problem (Holstein et al. 2019). We’re excited to support modeling practitioners in fairness-oriented analysis of models and look forward to seeing how these methods are put to work. References[](#references) -------------------------- Agarwal, Alekh, Alina Beygelzimer, Miroslav Dudı́k, John Langford, and Hanna Wallach. 2018. “A Reductions Approach to Fair Classification.” In _International Conference on Machine Learning_, 60–69. PMLR. Bell, Andrew, Lucius Bynum, Nazarii Drushchak, Tetiana Zakharchenko, Lucas Rosenblatt, and Julia Stoyanovich. 2023. “The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice.” In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_, 400–422. FAccT ‘23. New York, NY, USA: Association for Computing Machinery. [https://doi.org/10.1145/3593013.3594007](https://doi.org/10.1145/3593013.3594007) . Holstein, Kenneth, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. “Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?” In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_, 1–16. Mitchell, Shira, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021. “Algorithmic Fairness: Choices, Assumptions, and Definitions.” _Annual Review of Statistics and Its Application_ 8 (1): 141–63. [https://doi.org/10.1146/annurev-statistics-042720-125902](https://doi.org/10.1146/annurev-statistics-042720-125902) . Mittelstadt, Brent, Sandra Wachter, and Chris Russell. 2023. “The Unfairness of Fair Machine Learning: Levelling down and Strict Egalitarianism by Default.” _arXiv Preprint arXiv:2302.02404_. Narayanan, Arvind. 2018. “Translation Tutorial: 21 Fairness Definitions and Their Politics.” In _Proc. Conf. Fairness Accountability Transp., New York, Usa_, 1170:3. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Take the tidymodels survey for 2024 priorities [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Take the tidymodels survey for 2024 priorities ============================================== ![](/blog/2024/02/tidymodels-2024-survey/thumbnail-wd.jpg) Photo by [Aamyr](https://unsplash.com/photos/white-flowers-under-blue-sky-during-daytime-peN6l68AWaw)   2024/02/28   [tidymodels](/tags/tidymodels/) , [survey](/tags/survey/)   Emil Hvitfeldt At the end of 2021, we created a survey to get community input on how we prioritize our projects. [The results](https://colorado.posit.co/rsc/tidymodels-priorities-2022/) gave us a good sense of which items people were most interested in. Since then we have completed a number of projects: * **Model fairness metrics** were included in [yardstick 1.3.0](https://yardstick.tidymodels.org/news/index.html#yardstick-130) with [tidymodels.org](https://www.tidymodels.org/) posts coming soon. * **Spatial analysis models and methods** led to the creation of [spatialsample](https://spatialsample.tidymodels.org/) . * **H2O.ai support** was achieved with the creation of [agua](https://agua.tidymodels.org/) . * **Better serialization tools** are now provided in the [bundle](https://github.com/rstudio/bundle) package. Almost everything that respondents prioritized highly last year has either been completed or is currently in progress. Our main focus right now is to wrap up survival analysis, which is being done right now with a series of CRAN releases for the affected packages. Most immediately following these releases, we will be working on postprocessing and supervised feature selection. Beyond that, we’d like to once again ask the community for feedback to help us better prioritize features in the coming year. Looking toward 2024[](#looking-toward-2024) -------------------------------------------- **Take a look at [our survey for next priorities](https://conjoint.qualtrics.com/jfe/form/SV_aWw8ocGN5aPgeZE) ** and let us know what you think. There are some items we’ve put “on the menu” but you can write in other items that you are interested in. The current slate of our possible priorities include: ### Sparse tibbles[](#sparse-tibbles) Many models benefit from having sparse data, both in execution time and memory usage. We can’t take full advantage of this since recipes use tibbles. This project would involve making it so the tibbles used _inside of a recipe_ can hold sparse data. This would not be intended as a general substitute for regular tibbles. ### Causal inference interface[](#causal-inference-interface) While many common causal inference workflows are already possible with tidymodels, a small set of helper functions could greatly ease the experience of causal modeling in the framework. Specifically, these changes would better accommodate a two-stage modeling approach, using predictions from a propensity model to set case weights for an outcome model. ### Improve chattr[](#improve-chattr) [chattr](https://github.com/mlverse/chattr) is an interface to large language models (LLMs). It enables interaction with the model directly from the RStudio IDE. This task would involve fine-tuning it to give better results when used for tidymodels tasks. ### Cost-sensitive learning API[](#cost-sensitive-learning-api) This feature is another solution for severe class imbalances. The main part of this task is making our approaches to this uniform across models. ### Expand models for stacking ensembles[](#expand-models-for-stacking-ensembles) As of now, the stacks package only supports combining the predictions of member models using a regularized linear model. We could extend the package to allow for combining predictions using any modeling [workflow](https://workflows.tidymodels.org) . ### Extend support for spatial ML[](#extend-support-for-spatial-ml) [spatialsample](https://spatialsample.tidymodels.org/) introduced a number of spatial resampling methods to tidymodels. More comprehensive support for spatial ML would involve better integrating [spatial metrics](https://www.mm218.dev/posts/2022-08-11-waywiser-010-is-now-on-cran/) into the framework and introducing support for new spatial model types. ### Ordinal regression extension package[](#ordinal-regression-extension-package) Ordinal regression models are specific to classification tasks with a natural ordering to the outcome categories (e.g., low, medium, high, etc.). We could add support for modeling this type of data in a parsnip extension package. [Check out our survey](https://conjoint.qualtrics.com/jfe/form/SV_aWw8ocGN5aPgeZE) and tell us what your priorities are! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2 3.5.0: Introducing: coord_radial() [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) ggplot2 3.5.0: Introducing: coord\_radial() =========================================== ![](/blog/2024/03/ggplot2-3-5-0-coord-radial/thumbnail-wd.jpg) Photo by [Ismail Merad](https://unsplash.com/photos/ferris-wheel-beside-body-of-water-under-blue-sky-during-daytime-IWOo59NUXBk)   2024/03/01   [ggplot2](/tags/ggplot2/) , [ggplot2-3-5-0](/tags/ggplot2-3-5-0/)   Teun van den Brand We are happy to announce the release of [ggplot2](https://ggplot2.tidyverse.org) 3.5.0. This is one blogpost among several outlining a new polar coordinate system. Please find the [main release post](/blog/2024/02/ggplot2-3-5-0/) to read about other exciting changes. Polar coordinates are a good reminder of the flexibility of the Grammar of Graphics: pie charts are just bar charts with polar coordinates. While the tried and tested [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) has served well in the past to fulfill your pie chart needs, we felt it was due some modernisation. We realised we could not adapt [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) to fit with the [new guide system](/blog/2024/02/ggplot2-3-5-0/#guide-rewrite) without severely breaking existing plots, so [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) was born to give a facelift to the polar coordinate system in ggplot2. Relative to [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) , [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) can: 1. Draw circle sectors instead of only full circles. 2. Avoid data vanishing in the centre of the plot. 3. Adjust text angles on the fly. 4. Use the new guide system. An updated look[](#an-updated-look) ------------------------------------ The first noticeable contrast with [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) is that [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) is not particularly suited to building pie charts. Instead, it uses the scale expansion conventions like [`coord_cartesian()`](https://ggplot2.tidyverse.org/reference/coord_cartesian.html) . This makes sense for most chart types, but not pie charts. Nonetheless, you can use the `expand = FALSE` setting to use [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) for pie charts. library(ggplot2) library(patchwork) library(scales) pie <- ggplot(mtcars, aes(y = factor(1), fill = factor(cyl))) + geom_bar(width = 1) + scale_y_discrete(guide = "none", name = NULL) + guides(fill = "none") default <- pie + coord_radial() + ggtitle("default") no_expand <- pie + coord_radial(expand = FALSE) + ggtitle("expand = FALSE") polar <- pie + coord_polar() + ggtitle("coord_polar()") default | no_expand | polar ![Three pie charts showing the proportion of each cylinder number. The first has a gap in the middle and at the top with a grey circle in the background and is titled 'default'. The second is titled 'expand = FALSE' and shows a full pie chart with tick marks labelling the angle positions. The last plot is a full pie chart with a gray rectangular background without tick marks and a white line around the pie.](figs/compare_polar-1.png) Some visual differences stand out in the plots above. In [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) , the panel background covers the data area of the plot, not a rectangle. It also does not have a grid-line encircling the plot and instead uses tick marks to indicate values along the theta (angle) coordinate. You may also notice that [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) still draws the radius axis, despite instructions to use `guide = "none"`. That is the integration with the guide system that birthed [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) . Partial polar plots[](#partial-polar-plots) -------------------------------------------- Another important difference is that [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) does not necessarily need to display a full circle. By setting the `start` and `end` arguments separately, you can now make a partial polar plot. This makes it much easier to make semi- or quarter-circle plots. p <- ggplot(mpg, aes(displ, hwy)) + geom_point() half <- p + coord_radial(start = -0.5 * pi, end = 0.5 * pi) + ggtitle("−0.5π to +0.5π") quarter <- p + coord_radial(start = 0, end = 0.5 * pi) + ggtitle("0 to +0.5π") half | quarter ![Two polar scatterplots of the 'mpg' dataset. The left plot is shaped like as a semicircle and the right plot as a quarter circle.](figs/partial_polar-1.png) Donuts[](#donuts) ------------------ It was already possible to turn a pie-chart into a donut-chart with [`coord_polar()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) . This is made even easier in [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) by setting the `inner.radius` argument to make a donut hole. For most plots, this avoids crowding data points in the center of the plot: points with a widely different `theta` coordinate but similarly small `r` coordinate are placed further apart. p + coord_radial(inner.radius = 0.3, r_axis_inside = TRUE) ![A donut-shaped scatterplot of the 'mpg' dataset.](figs/open_radial_plot-1.png) Text annotations[](#text-annotations) -------------------------------------- A common grievance with about polar coordinates is that it was cumbersome to rotate text annotations along with the `theta` coordinate. Calculating the correct angles for labels is pretty involved and usually changes from plot to plot depending on how many items need to be displayed. To remove some of this hassle [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) has a `rotate_angle` switch, that will line up the text’s `angle` aesthetic with the theta coordinate. For text angles of 0 degrees, this will place text in a tangent orientation to the circle and for angles of 90 degrees, this places text along the radius, as in the plot below. ggplot(mtcars, aes(seq_along(mpg), mpg)) + geom_col(width = 1) + geom_text( aes(y = 32, label = rownames(mtcars)), angle = 90, hjust = 1 ) + coord_radial(rotate_angle = TRUE, expand = FALSE) ![A wind rose plot showing miles per gallon for different cars. The car names skirt the outer edge of the plot and are oriented towards the centre.](figs/text_angles-1.png) Axes[](#axes) -------------- Because the logic of drawing axes for polar coordinates is not the same as when axes are perfectly vertical or horizontal, we used the new guide system to build an axis specific to [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) : the [`guide_axis_theta()`](https://ggplot2.tidyverse.org/reference/guide_axis_theta.html) axis. Guides for [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) can be set using `theta` and `r` name in the [`guides()`](https://ggplot2.tidyverse.org/reference/guides.html) function. While the `r` axis can be the regular [`guide_axis()`](https://ggplot2.tidyverse.org/reference/guide_axis.html) , the `theta` axis uses the highly specialised [`guide_axis_theta()`](https://ggplot2.tidyverse.org/reference/guide_axis_theta.html) . The theta axis shares many features with typical axes, like setting the text angle or the new `minor.ticks` and `cap` settings. More on these settings in the [axis blog](/blog/2024/02/ggplot2-3-5-0-axes/) . As seen in previous plots, the default is to place text horizontally. One neat trick we’ve put into [`coord_radial()`](https://ggplot2.tidyverse.org/reference/coord_polar.html) is that we can set a _relative_ text angle in the guides, such as in the plot below. ggplot(mpg, aes(class, displ)) + geom_boxplot() + coord_radial(start = 0.25 * pi, end = 1.75 * pi) + guides( theta = guide_axis_theta(angle = 0), r = guide_axis(angle = 0) ) ![Boxplot of the 'mpg' dataset displayed in partial polar coordinates. The theta labels are placed tangential to the circle. The radius labels line up with the tick mark direction.](figs/axis_angles-1.png) The theme elements to style these axes have the `theta` or `r` position indication, so to change the the axis line, you use the `axis.line.theta` and `axis.line.r` arguments. The theme settings can also be used to set the _absolute_ angle of text. ggplot(mpg, aes(class, displ)) + geom_boxplot() + coord_radial(start = 0.25 * pi, end = 1.75 * pi) + theme( axis.line.theta = element_line(colour = "red"), axis.text.theta = element_text(angle = 90), axis.text.r = element_text(colour = "blue") ) ![Boxplot of the 'mpg' dataset displayed in partial polar coordinates. The theta labels are placed vertically and a red line traces the outer circle. The radius labels are displayed in blue.](figs/axis_styling-1.png) Lastly, there can also be secondary axes. We anticipate that this is practically never needed, as grid lines follow the primary axes and without them, it is very hard to read from axes in polar coordinates. However, if there is some reason for using secondary axes on polar coordinates, you can use the `theta.sec` and `r.sec` names in the [`guides()`](https://ggplot2.tidyverse.org/reference/guides.html) function to control the guides. Please note that a secondary theta axis is entirely useless when `inner.radius = 0` (the default). There are no separate theme options for secondary r/theta axes, but to style them separately from the primary axes, you can use the `theme` argument in the guide instead. ggplot(pressure, aes(temperature, pressure)) + geom_line(colour = "blue") + scale_x_continuous( labels = label_number(suffix = "°C"), sec.axis = sec_axis(~ .x * 9/5 + 35, labels = label_number(suffix = "°F")) ) + scale_y_continuous( labels = label_number(suffix = " mmHg"), sec.axis = sec_axis(~ .x * 0.133322, labels = label_number(suffix = " kPa")) ) + guides( theta.sec = guide_axis_theta(theme = theme(axis.line.theta = element_line())), r.sec = guide_axis(theme = theme(axis.text.r = element_text(colour = "red"))) ) + coord_radial( start = 0.25 * pi, end = 1.75 * pi, inner.radius = 0.3 ) ![A lineplot of the 'pressure' dataset in partial polar coordinates that is shaped like a donut with a bite taken out on top. The primary, outer theta axis displays temperature in degrees Celcius. The secondary, inner theta axis displays temperature in degrees Fahrenheit and has an axis line. The primary radius axis on the right displays pressure in millimetres of mercury. The secondary radius axis on the left displays pressure in kilo-Pascals in red text.](figs/secondary_axes-1.png) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # webR 0.3.1 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) webR 0.3.1 ========== ![](/blog/2024/04/webr-0-3-1/thumbnail-wd.jpg) Photo by [Sandro Katalina](https://unsplash.com/photos/purple-and-black-pyramid-wallpaper-k1bO_VTiZSs)   2024/04/02   [webr](/tags/webr/) , [wasm](/tags/wasm/) , [webassembly](/tags/webassembly/)   George Stagg We’re delighted to announce the release of [webR](https://docs.r-wasm.org/webr/latest/) 0.3.1. This release brings bug fixes, infrastructure upgrades, and exciting improvements to webR’s API for creating R objects and evaluating R code from JavaScript. These new features make integrating webR with existing JavaScript frameworks such as [Observable](https://observablehq.com) a breeze. You can install the latest release from [npm](https://www.npmjs.com/package/webr) with the command: npm i webr@0.3.1 or if you’re using JavaScript modules to import webR directly from CDN: import { WebR } from 'https://webr.r-wasm.org/v0.3.1/webr.mjs'; A summary of changes is described below, with the full [release notes](https://github.com/r-wasm/webr/releases) on GitHub. Evaluating R code from JavaScript[](#evaluating-r-code-from-javascript) ------------------------------------------------------------------------ The underlying interpreter powering webR is built from the same source code as R itself, with patches applied so that it can run in the WebAssembly environment. With this release, we have rebased our patches on the latest stable version of R[1](#fn:1) . By keeping our source in sync, improvements and bug fixes made by the R Core Team also benefit any project making use of webR. WebR’s core functionality is to evaluate R code from a JavaScript environment. As such, it is imperative that this works well, even with large and complex scripts. The [webR app](https://webr.r-wasm.org/v0.3.1/) has been updated to better handle large R scripts, and scripts longer than 4096 characters should no longer cause strange issues in the R console. ### Loading WebAssembly packages[](#loading-webassembly-packages) The package management functions provided by webR have been expanded and improved. We set up webR with shims (interceptors) for [`install.packages()`](https://rdrr.io/r/utils/install.packages.html) , [`library()`](https://rdrr.io/r/base/library.html) , and [`require()`](https://rdrr.io/r/base/library.html) so that installing or loading R packages automatically downloads WebAssembly binaries from the [webR package repository](https://repo.r-wasm.org) . Also, it is no longer required to run the [`library()`](https://rdrr.io/r/base/library.html) command a second time to subsequently load the package. In this interactive example, webR is configured to automatically install WebAssembly packages. Click “Run code” to download the packages listed in the R script. Loading webR... See the [documentation](https://docs.r-wasm.org/webr/latest/packages.html) for more details on how to control this behaviour in your own webR-powered applications, including optionally showing an interactive download menu to the user. ### Error handling and reporting[](#error-handling-and-reporting) Improvements have been made to how webR raises R conditions as JavaScript exceptions. Exceptions now include the offending source R call in the error message text, better matching what is shown in a traditional R console. await webR.evalR("sin('abc')"); Uncaught Error: Error in sin("abc"): non-numeric argument to mathematical function Conditions raised when invoking function objects are now also re-thrown as JavaScript exceptions, rather than a generic `UnwindProtectException` error. Compare the error messages shown below from the previous and latest versions of webR to see the useful context added by this change. // webR 0.2.2 const do_calc = await webR.evalR(`function (n) { rnorm(n) }`) do_calc(-10) Uncaught (in promise) UnwindProtectException: A non-local transfer of control occured during evaluation // webR 0.3.1 const do_calc = await webR.evalR(`function (n) { rnorm(n) }`) do_calc(-10) Uncaught (in promise) Error: Error in rnorm(n): invalid arguments Some base R features can be problematic when running R under WebAssembly. For example, in the constrained WebAssembly sandbox the base R function [`system()`](https://rdrr.io/r/base/system.html) does not work. The latest release of webR now handles these cases more consistently, raising R [`stop()`](https://rdrr.io/r/base/stop.html) conditions rather than incorrectly returning an empty result. # webR 0.3.1 system() Error in webr_hook_system(command) : The "system()" function is unsupported under Emscripten. Capturing HTML canvas graphics output[](#capturing-html-canvas-graphics-output) -------------------------------------------------------------------------------- The [`captureR()`](https://docs.r-wasm.org/webr/latest/evaluating.html#evaluating-r-code-and-capturing-output-with-capturer) function is designed to capture output generated when evaluating R code. In addition to capturing standard text output, details about errors and other R conditions are also captured. With this release, plots drawn using webR’s HTML canvas graphics device, [`webr::canvas()`](https://rdrr.io/pkg/webr/man/canvas.html) , are also captured and returned by default. // Evaluate R code, capturing all output const capture = await webR.globalShelter.captureR(` x <- rnorm(10000) print(x[1]) hist(x) `); console.log(capture); { result: Proxy(Object), output: [\ { type: 'stdout', data: '[1] 0.7612882' },\ ], images: [ ImageBitmap ], } Captured plots are returned as an array of [`ImageBitmap`](https://developer.mozilla.org/en-US/docs/Web/API/ImageBitmap) JavaScript objects in the `images` property. This interface represents a bitmap image in a way that can be efficiently drawn to a HTML [``](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/canvas) element. This change makes plotting consistent with other forms of R output and simplifies the process when working with multiple independent R code blocks and output images. See the webR documentation on [evaluating R code](https://docs.r-wasm.org/webr/latest/evaluating.html#evaluating-r-code-and-capturing-output-with-capturer) for further details, and this [Observable notebook](https://observablehq.com/d/ec99bb89a4c646ab) for an example of capturing R plots from JavaScript. ### Graphics device bug fixes[](#graphics-device-bug-fixes) In addition to adding the ability to capture graphics output, the [`webr::canvas()`](https://rdrr.io/pkg/webr/man/canvas.html) graphics device has also had various bug fixes made to better implement R base graphics. The easiest way to demonstrate is probably by example: Loading webR... Loading webR... Loading webR... The R object interface[](#the-r-object-interface) -------------------------------------------------- The R object interface provided by webR has been expanded to support the conversion of more types of JavaScript objects into R objects. Such conversions are automatically applied when interacting with the R environment from JavaScript. ### Raw vectors[](#raw-vectors) JavaScript objects of type `TypedArray`, `ArrayBuffer`, and `ArrayBufferView` (e.g.  [`Uint8Array`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) ) may now be used to construct R objects. By default, objects of this type are converted to R raw atomic vectors. This simplifies the transfer of binary data to R. const data = new Uint8Array([4, 12, 8, 24, 15, 12]); // Print data's R object class and an example byte await webR.evalR(` class(data) data[2] `, { withAutoprint: true, env: { data } }); [1] "raw" [1] 0c ### R `data.frame`[](#r-dataframe) JavaScript objects of shape `{ x: [...], y: [...] }`, with data in a “long” column-based form, can now be used to construct R objects. In previous versions of webR, this object shape was reserved for future use. However, with this release webR now constructs an R `data.frame` by taking the source object’s properties as column vectors. The resulting `data.frame` can then be manipulated from R in the usual way: const data = { column_x: ["foo", "bar", "baz"], column_y: [1, 3, 7] } await webR.evalR(` class(data) colnames(data) data[2:3,] `, { withAutoprint: true, env: { data } }); [1] "data.frame" [1] "column_x" "column_y" column_x column_y 2 bar 3 3 baz 7 Similarly, an R `data.frame` can be converted back into a JavaScript object of this form: const cars = await webR.evalR(`mtcars`); await cars.toObject(); { am: [1, 1, 1, ..., 1], carb: [4, 4, 1, ..., 2], cyl: [6, 6, 4, ..., 4] ..., wt: [2.62, 2.875, 2.32, ..., 2.78], } ### D3 “wide” format[](#d3-wide-format) In JavaScript, particularly when using frameworks built upon [D3](https://d3js.org) , it is typical to work with data in a “wide” form: an array of objects per row, each including all the column names and values. With this release, webR can also convert JavaScript objects in this form into an R `data.frame`. The following example loads the same data as shown in the previous example but expressed in the “wide” form. const data = [\ { column_x: "foo", column_y: 1 },\ { column_x: "bar", column_y: 3 },\ { column_x: "baz", column_y: 7 },\ ]; await webR.evalR(` class(data) colnames(data) data[2:3,] `, { withAutoprint: true, env: { data } }); [1] "data.frame" [1] "column_x" "column_y" column_x column_y 2 bar 3 3 baz 7 An R `data.frame` can also be converted into a D3 compatible JavaScript object: const cars = await webR.evalR(`mtcars`); await cars.toD3(); [\ { mpg: 21, cyl: 6, disp: 160, ... },\ { mpg: 21, cyl: 6, disp: 160, ... },\ { mpg: 22.8, cyl: 4, disp: 108, ...},\ ...\ { mpg: 21.4, cyl: 4, disp: 121, ...},\ ] WebAssembly toolchain upgrades[](#webassembly-toolchain-upgrades) ------------------------------------------------------------------ We have updated our WebAssembly build system, upgrading the [Emscripten](https://emscripten.org) C/C++ compiler to version 3.1.47 and the [LLVM Flang](https://flang.llvm.org/docs/) Fortran compiler to be based on LLVM 18.1.1. As part of the work, webR now supports building under Nix using [flakes](https://nixos.wiki/wiki/Flakes) , suggested and largely implemented by [@wch](https://github.com/wch) . With this, source-code level reproducible builds of the webR WebAssembly binaries can be made, strengthening the argument for webR as a potential future platform for reproducible data science. ### LLVM Flang[](#llvm-flang) To compile Fortran sources in the R source code[2](#fn:2) for webR, we require a Fortran compiler that supports outputting WebAssembly objects. This is a surprisingly tricky business, and our current solution is to maintain a patched version of LLVM’s `flang-new` compiler frontend. In recent months, the patches we must make to LLVM Flang have become smaller and easier to manage as the LLVM team continues to improve the Flang frontend. While too long for this post, for those interested in exactly what changes we make to enable WebAssembly output, I have written a deep-dive blog post, [Fortran on WebAssembly](https://gws.phd/posts/fortran_wasm/) . Additional system libraries and Rust support[](#additional-system-libraries-and-rust-support) ---------------------------------------------------------------------------------------------- Thanks to some great work by [@jeroen](https://github.com/jeroen) and [@yutannihilation](https://github.com/yutannihilation) , this release of webR includes some additional WebAssembly system libraries and software in the webR Docker container. This includes numerical libraries such as [GSL](https://www.gnu.org/software/gsl/) and [GMP](https://gmplib.org) , image manipulation tools such as [ImageMagick](https://imagemagick.org/) , and a Rust compiler configured to build WebAssembly R packages containing Rust source code. A demonstration R package containing Rust code, compatible with webR, can be found at [https://github.com/yutannihilation/savvy-webr-test/](https://github.com/yutannihilation/savvy-webr-test/) . An example Shiny app making use of the WebAssembly compiled ImageMagick library is shown below, with the source code at [https://github.com/jeroen/shinymagick](https://github.com/jeroen/shinymagick) . WebAssembly R package binaries[](#webassembly-r-package-binaries) ------------------------------------------------------------------ With the introduction of additional system libraries and changes to the WebAssembly toolchain, the default webR package repository has also been refreshed. The repository tends to follow CRAN package releases, though is updated less frequently. **19452** WebAssembly R packages have been recompiled from source for this release, with **12969** packages, about 63% of CRAN, fully available[3](#fn:3) for use in webR. As my usual caveat goes, we have not been able to test all the available packages. Feel free to try your favourite package in the [webR app](https://webr.r-wasm.org/v0.3.1/) and let us know in a [GitHub issue](https://github.com/r-wasm/webr/issues) if there is a problem. The [package repository index](https://repo.r-wasm.org) contains further information and a searchable list of WebAssembly R packages. In addition, [R-Universe](https://r-universe.dev) also builds webR-compatible binaries and so can be used as an alternative repository for access to even more R packages. ### Building custom WebAssembly R packages[](#building-custom-webassembly-r-packages) If you’d like to build your own R packages for webR, the [rwasm](https://r-wasm.github.io/rwasm/) package provides functions to help compile R packages for WebAssembly, manage repositories, and prepare webR-compatible filesystem images. We’ve also started building [reusable workflows for GitHub Actions](https://github.com/r-wasm/actions/) . If you have an R package with source code hosted on GitHub, an action can be added to your repository such that a WebAssembly version of your package will be built automatically by a GitHub runner on package release. Acknowledgements[](#acknowledgements) -------------------------------------- Thank you, as always, to the users and developers contributing to webR in the form of discussion in issues, bug reports, and pull requests. [@adrianolszewski](https://github.com/adrianolszewski) , [@christianp](https://github.com/christianp) , [@coatless](https://github.com/coatless) , [@ColinFay](https://github.com/ColinFay) , [@drgomulka](https://github.com/drgomulka) , [@erex](https://github.com/erex) , [@gitdemont](https://github.com/gitdemont) , [@gorkang](https://github.com/gorkang) , [@isbool](https://github.com/isbool) , [@JeremyPasco](https://github.com/JeremyPasco) , [@jeroen](https://github.com/jeroen) , [@JosiahParry](https://github.com/JosiahParry) , [@Luke-Symes-Tsy](https://github.com/Luke-Symes-Tsy) , [@maek-ies](https://github.com/maek-ies) , [@MaybeJustJames](https://github.com/MaybeJustJames) , [@ravinder387](https://github.com/ravinder387) , [@StaffanBetner](https://github.com/StaffanBetner) , [@SugarRayLua](https://github.com/SugarRayLua) , [@takahser](https://github.com/takahser) , [@tim-newans](https://github.com/tim-newans) , [@timelyportfolio](https://github.com/timelyportfolio) , [@tstubbs-evolution](https://github.com/tstubbs-evolution) , [@yhm-amber](https://github.com/yhm-amber) , [@yii-iiy](https://github.com/yii-iiy) , [@yutannihilation](https://github.com/yutannihilation) , and [@zhangwenda0518](https://github.com/zhangwenda0518) . * * * 1. The latest stable release at the time of writing: [R 4.3.3 — “Angel Food Cake”](https://cran.rstudio.com/doc/manuals/r-release/NEWS.html) [↩︎](#fnref:1) 2. There are also many R packages containing Fortran source code. [↩︎](#fnref:2) 3. Here “available” means that both a binary build of an R package and all of its dependencies can be downloaded from the repository. [↩︎](#fnref:3) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2-3-5-0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Ggplot2-3-5-0 [![](/blog/2024/03/ggplot2-3-5-0-coord-radial/thumbnail-sq.jpg)](/blog/2024/03/ggplot2-3-5-0-coord-radial/) [ggplot2 3.5.0: Introducing: coord\_radial()](/blog/2024/03/ggplot2-3-5-0-coord-radial/) [package](/categories/package) Teun van den Brand Introducing a new polar coordinate system that supersedes the old `coord_polar()`. Read on about the new `coord_radial()`. [Read more ...](/blog/2024/03/ggplot2-3-5-0-coord-radial/) 2024/03/01 [![](/blog/2024/02/ggplot2-3-5-0-axes/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0-axes/) [ggplot2 3.5.0: Axes](/blog/2024/02/ggplot2-3-5-0-axes/) [package](/categories/package) Teun van den Brand The 3.5.0 version of ggplot2 comes with an overhaul of the guide system. Read here what is new for axes. [Read more ...](/blog/2024/02/ggplot2-3-5-0-axes/) 2024/02/28 [![](/blog/2024/02/ggplot2-3-5-0-legends/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0-legends/) [ggplot2 3.5.0: Legends](/blog/2024/02/ggplot2-3-5-0-legends/) [package](/categories/package) Teun van den Brand The 3.5.0 version of ggplot2 comes with an overhaul of the guide system. Read here what is new for legends. [Read more ...](/blog/2024/02/ggplot2-3-5-0-legends/) 2024/02/26 [![](/blog/2024/02/ggplot2-3-5-0/thumbnail-sq.jpg)](/blog/2024/02/ggplot2-3-5-0/) [ggplot2 3.5.0](/blog/2024/02/ggplot2-3-5-0/) [package](/categories/package) Teun van den Brand ggplot2 3.5.0 is now on CRAN. Discover what is new in this release. [Read more ...](/blog/2024/02/ggplot2-3-5-0/) 2024/02/23 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2 3.5.0: Legends [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) ggplot2 3.5.0: Legends ====================== ![](/blog/2024/02/ggplot2-3-5-0-legends/thumbnail-wd.jpg) Photo by [Markus Spiske](https://unsplash.com/photos/close-up-photo-of-black-camera-lens-hqCEQTc5gZA)   2024/02/26   [ggplot2](/tags/ggplot2/) , [ggplot2-3-5-0](/tags/ggplot2-3-5-0/)   Teun van den Brand We are pleased to release [ggplot2](https://ggplot2.tidyverse.org) 3.5.0. This is one blogpost among several outlining changes to legend guides. Please find the [main release post](/blog/2024/02/ggplot2-3-5-0/) to read about other changes. Legends, alongside axes, are visual representations of scales and allow observes to translate graphical properties of a plot into information. To no surprise, legends in ggplot2 comprise the guides called [`guide_legend()`](https://ggplot2.tidyverse.org/reference/guide_legend.html) , but also [`guide_colourbar()`](https://ggplot2.tidyverse.org/reference/guide_colourbar.html) , [`guide_coloursteps()`](https://ggplot2.tidyverse.org/reference/guide_coloursteps.html) and [`guide_bins()`](https://ggplot2.tidyverse.org/reference/guide_bins.html) . Styling[](#styling) -------------------- One of the more user-visible changes is that these guides no longer have styling options. Or at least, they have been soft-deprecated: they continue to work for now, but are scheduled for removal. Gone are the days where there were 4 possible ways to set the horizontal justification of legend text in 5 different functions. There is only one way to style guides now, and that is by using [`theme()`](https://ggplot2.tidyverse.org/reference/theme.html) . The [`theme()`](https://ggplot2.tidyverse.org/reference/theme.html) function has new arguments to control the appearance of legends, which makes it easier to globally control the appearance of legends. For example: `theme(legend.frame)` replaces `guide_colourbar(frame.colour, frame.linewidth, frame.linetype)` and `theme(legend.axis.line)` replaces `guide_bins(axis, axis.colour, axis.linewidth, axis.arrow)`. To allow for tweaking the style of any individual guide, the guide functions now have a `theme` argument that can accept a theme specific to that guide. library(ggplot2) ggplot(mpg, aes(displ, hwy, shape = factor(cyl), colour = cty)) + geom_point() + # Styling individual guides guides( shape = guide_legend(theme = theme(legend.text = element_text(colour = "red"))), colour = guide_colorbar(theme = theme(legend.frame = element_rect(colour = "red"))) ) + # Styling guides globally theme( legend.title.position = "left", # Title justification is controlled by hjust/vjust in the element legend.title = element_text(angle = 90, hjust = 0.5) ) ![Scatterplot of engine displacement versus highway miles per gallon. The legend indicating shapes for the number of cylinders has red text. The colour bar indicating city miles per gallon has a red rectangle around the bar. Both the legend and colour bar titles are rotated, centered and on the left of the guide.](figs/guide_theming-1.png) In the plot above, notice how the legend title settings affect both the colour bar and the legend, whereas the local options, like red legend text, only apply to a single guide. Awareness[](#awareness) ------------------------ Legends are now more aware what discrete variables should be placed in which keys. By default, they now only draw keys for the layer which contain the relevant value. This saves one having to hassle with the `guide_legend(override.aes)` argument to get the keys to display just right. In the plot below, notice how the points and line have separate keys. p <- ggplot(mpg, aes(displ, hwy)) + scale_alpha_manual(values = c(0.5, 1)) p + geom_point(aes(colour = "points", alpha = "points")) + geom_line( aes(colour = "line", alpha = "line"), stat = "smooth", formula = y ~ x, method = "lm" ) ![A scatterplot with trendline showing engine displacement versus highway miles per gallon. There are two legends for colour and alpha. Both legends show points and lines separately.](figs/legend_awareness-1.png) To revert back to the old behaviour, you can set the `show.legend = TRUE` option in the layers. Like before, the `show.legend` argument can still be set in an aesthetic-specific way. Setting it to `TRUE` means ‘always show’, `FALSE` means ‘never show’ and `NA` means ‘show if found’. p + geom_point( aes(colour = "points", alpha = "points"), show.legend = TRUE # always show ) + geom_line( aes(colour = "line", alpha = "line"), stat = "smooth", formula = y ~ x, method = "lm", show.legend = c(colour = NA, alpha = TRUE) # always show in alpha ) ![The same plot as before, but every legend keys displays points. Lines are shown in every 'alpha' legend key, but only one 'colour' key.](figs/show_key_setting-1.png) Placement[](#placement) ------------------------ Legend positions are no longer restricted to just a single side of the plot. By setting the `position` argument of guides, you can tailor which guides appear where in the plot. Guides that do not have a position set, like the ‘drv’ shape legend below, follow the global theme’s `legend.position` setting. If we suspend our belief in good data visualisation practice, we can showcase this as follows: p <- ggplot(mpg, aes(displ, hwy, shape = drv, colour = cty, size = year)) + geom_point(aes(alpha = cyl)) + guides( colour = guide_colourbar(position = "bottom"), size = guide_legend(position = "top"), alpha = guide_legend(position = "inside") ) + theme(legend.position = "left") p ![A scatterplot showing engine displacement versus highway miles per gallon. It has four legend placed at the top, left, bottom of the panel and one inside the panel.](figs/legend_positions-1.png) In the plot above, the legend for the ‘cyl’ variable is in the middle of the plot. In previous versions of ggplot2, you could set the `legend.position` to a coordinate to control the placement. However, doing this would change the default legend position, which is not always desirable. To cover such cases, there is now a specialised `legend.position.inside` argument that controls the positioning of legends with `position = "inside"` regardless of whether the position was specified in the theme or in the guide. p + theme(legend.position.inside = c(0.7, 0.7)) ![The same plot as before, but the legend for the 'cyl' variable is to the top-right of the centre.](figs/legend_inside-1.png) The justification of legends is controllable by using the `legend.justification.{position}` theme setting. Moreover, the top and bottom guides can be aligned to the plot rather than the panel by setting the `legend.location` argument. The main reason behind this is that you can then align the legends with the plot’s title. By default, when `plot.title.position = "plot"`, left legends are already aligned. For this reason, the top and bottom guides are prioritised for the `legend.location` setting. Moreover, it avoids overlapping of legends in the corners if the justifications would dictate it. p + labs(title = "Plot-aligned title") + theme( legend.margin = margin(0, 0, 0, 0), # turned off for alignment legend.justification.top = "left", legend.justification.left = "top", legend.justification.bottom = "right", legend.justification.inside = c(1, 1), legend.location = "plot", plot.title.position = "plot" ) ![The same plot as before, but with a plot-aligned title and different alignments of the legends. The left and top legends are left-aligned with the title.](figs/legend_alignments-1.png) Spacing and margins[](#spacing-and-margins) -------------------------------------------- In this release, the way spacing in legends work has been reworked. * The `legend.spacing{.x/.y}` theme setting is now used to space different guides apart. Previously, it was also used to space legend keys apart; that is no longer the case. * Spacing legend key-label pairs apart is now controlled by the `legend.key.spacing{.x/.y}` theme setting. * Spacing the labels from the keys is now controlled by the label element’s `margin` argument. Because the legend spacing and margin options can be a bit bewildering, a small overview is added below. One setting not included in the overview is `legend.spacing.x`, which only applies when `legend.box = "horizontal"`. Which exact text margin is relevant for spacing apart keys and labels, or titles and the rest of the guide, depends on the `legend.text.position` and `legend.title.position` theme elements. ![Overview of legend spacing and margin options. Two abstract legends are placed above one another to the right of an area called 'plot'. Various arrows with labels point out different theme settings.](figs/spacing_overview-1.png) When the titles and keys don’t have explicit margins, appropriate margins are added automatically depending on the text or title position. However, if you override the margins, they will be interpreted literally. ggplot(mpg, aes(displ, hwy, colour = class)) + geom_point() + guides(colour = guide_legend(ncol = 2)) + theme( legend.key.spacing.x = unit(10, "pt"), legend.key.spacing.y = unit(20, "pt"), legend.text = element_text(margin = margin(l = 0)), legend.title = element_text(margin = margin(b = 20)) ) ![A scatterplot showing engine displacement versus highway miles per gallon. The legend for the 'class' variable shows a key layout with two columns. Keys are widely spacing in the vertical direction and more narrowly in the horizontal direction. There is no space between the keys and their labels, but plenty of space between the legend and its title.](figs/legend_spacing-1.png) For all intents and purposes, colour bar/step and bins guides are treated as legend guides with just a single key-label pair. While the `legend.key.spacing` setting does not apply due to it being one single key, the other spacings and margins do apply equally. ggplot(mpg, aes(displ, hwy, colour = cty)) + geom_point() + theme( legend.text = element_text(margin = margin(l = 0)), legend.title = element_text(margin = margin(b = 20)) ) ![The same plot as before, but with a colourbar indicating the 'cty' variable. Again, there is no space between the bar and the labels and ample space between the bar and the title.](figs/legend_spacing_bar-1.png) Stretching[](#stretching) -------------------------- Another experimental tweak to legends is that they can now have stretching keys (or bars). The option is still considered ‘experimental’ because there are some things that may go wrong. By setting the `legend.key{.height/.width}` theme argument as a `"null"` unit, legends can now expand to fill the available space. p <- ggplot(mpg, aes(displ, hwy)) + geom_point(aes(colour = cty, size = cyl), shape = 21) + theme(legend.key.height = unit(1, "null")) p ![Scatterplot of engine displacement versus highway miles per gallon. There is a legend guide showing the point's size and a colour. Both the legend and the bar take up an approximately equal amount of space on the right-hand side of the panel.](figs/stretch_keys-1.png) The term ‘available space’ is a tricky one. For starters, other legends placed in the same position take up space, as can be seen in the plot above. If your legend is the only legend in a position, more space is available and it stretches more. As you can see in the plot below, the legends are not aligned with the panel even when stretched. This is because the titles, margins and various spacings all take up space that is _not_ available to stretch into. p + guides(colour = guide_colourbar(position = "left")) ![Same plot as before, but the colour bar is placed on the left. Both the colour bar and legend take up a lot of vertical space.](figs/isolated_stretch-1.png) On the other hand, if one position is packed with legends, the keys may shrink instead of stretch. The keys can become too small to show the aesthetics properly. You can see in the example below that the size legend becomes cut-off due to small keys and text is spaced too closely to comfortably read. p + aes(fill = model) ![Same plot as before, but all legends are on the right, including a new legend for the 'model' variable. All legends have keys that are too small to read the text comfortably, and the points indicating size are clipped.](figs/shrinking_keys-1.png) Another issue that may come up is that the ‘available space’ might be 0. Because the plot itself is also space-filling, setting null-heights for top/bottom positions or null-widths for left/right positions means there is no available space. This may result in the keys or bars becoming invisible. For the plot below, recall that we’ve set the `legend.key.height` setting to a null unit. p + theme(legend.position = "top") ![Still the same scatterplot but without the fill variable. Legends are placed at the top of the panel, but the bar and key backgrounds have disappeared. The text labels are still present.](figs/disappearing_keys-1.png) Other improvements[](#other-improvements) ------------------------------------------ We welcome a new type of legend: [`guide_custom()`](https://ggplot2.tidyverse.org/reference/guide_custom.html) . It can be used to add any graphical object (grob) to a plot, like [`annotation_custom()`](https://ggplot2.tidyverse.org/reference/annotation_custom.html) . There are a few differences though: it is positioned just like a legend and adds titles and margins. In some sense, this guide is ‘special’, as it is the only guide that does not directly reflect a scale. The downside is that it cannot read properties from the plot, but the upside is that it is very flexible. Be careful when your grob does not have an absolute size, you should set the `width` and `height` arguments. x <- c(0.5, 1, 1.5, 1.2, 1.5, 1, 0.5, 0.8, 1, 1.15, 2, 1.15, 1, 0.85, 0, 0.85) y <- c(1.5, 1.2, 1.5, 1, 0.5, 0.8, 0.5, 1, 2, 1.15, 1, 0.85, 0, 0.85, 1, 1.15) compass_rose <- grid::polygonGrob( x = unit(x, "cm"), y = unit(y, "cm"), id.lengths = c(8, 8), gp = grid::gpar(fill = c("grey50", "grey25"), col = NA) ) nc <- sf::st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE) ggplot(nc) + geom_sf(aes(fill = AREA)) + guides(custom = guide_custom(compass_rose, title = "compass")) ![A map of the US state North Carolina, where fill colour indicates the area of counties. Underneath the colour bar for the fill, there is an eight-pointed star to the right of the panel with the title 'compass'.](figs/custom_guide-1.png) In previous version of ggplot2, when legend titles are wider than the legends, the guide-title alignment was always left aligned. Now, the justification setting of the legend text determines the alignment: 1 is right or top aligned and 0 is left or bottom aligned. ggplot(mpg, aes(displ, hwy, shape = factor(cyl), colour = drv)) + geom_point() + guides( shape = guide_legend( title = "A title that is pretty long", theme = theme(legend.title = element_text(hjust = 1)), order = 1 ), colour = guide_legend( title = "Another long title", theme = theme(legend.title = element_text(hjust = 0)) ) ) ![Scatterplot of engine displacement versus highway miles per gallon. The 'drv' variable has a legend that is left aligned, whereas the 'cyl' variable has a legend that is right-aligned.](figs/title_justification-1.png) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # survey [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Survey [![](/blog/2024/02/tidymodels-2024-survey/thumbnail-sq.jpg)](/blog/2024/02/tidymodels-2024-survey/) [Take the tidymodels survey for 2024 priorities](/blog/2024/02/tidymodels-2024-survey/) [other](/categories/other) Emil Hvitfeldt We are conducting our third tidymodels priorities survey. Please give us your feedback! [Read more ...](/blog/2024/02/tidymodels-2024-survey/) 2024/02/28 [![](/blog/2021/12/tidymodels-2021-q4/thumbnail-sq.jpg)](/blog/2021/12/tidymodels-2021-q4/) [Closing out our year with a Q4 2021 tidymodels update](/blog/2021/12/tidymodels-2021-q4/) [roundup](/categories/roundup) Julia Silge New features like an R Markdown template and Shiny app offer more robust support for modeling analyses, and our survey results outline some priorities for next year. [Read more ...](/blog/2021/12/tidymodels-2021-q4/) 2021/12/16 [![](/blog/2021/10/tidymodels-2022-survey/thumbnail-sq.jpg)](/blog/2021/10/tidymodels-2022-survey/) [Take the tidymodels survey for 2022 priorities](/blog/2021/10/tidymodels-2022-survey/) [other](/categories/other) Max Kuhn We are conducting our second tidymodels priorities survey. Please give us your feedback! [Read more ...](/blog/2021/10/tidymodels-2022-survey/) 2021/10/06 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # bigrquery [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Bigrquery [![](/blog/2024/01/bigrquery-1-5-0/thumbnail-sq.jpg)](/blog/2024/01/bigrquery-1-5-0/) [bigrquery 1.5.0](/blog/2024/01/bigrquery-1-5-0/) [package](/categories/package) Hadley Wickham This release fixes a bunch of annoyances and catches up with innovations in DBI and dbplyr. [Read more ...](/blog/2024/01/bigrquery-1-5-0/) 2024/01/22 [![](/blog/2021/08/bigrquery-1-4-0/thumbnail-sq.jpg)](/blog/2021/08/bigrquery-1-4-0/) [bigrquery 1.4.0](/blog/2021/08/bigrquery-1-4-0/) [package](/categories/package) Jenny Bryan bigrquery 1.4.0 fixes a bug in `bq_table_download()`. [Read more ...](/blog/2021/08/bigrquery-1-4-0/) 2021/08/04 [![](/blog/2021/07/gargle-1-2-0/thumbnail-sq.jpg)](/blog/2021/07/gargle-1-2-0/) [gargle 1.2.0](/blog/2021/07/gargle-1-2-0/) [package](/categories/package) Jenny Bryan gargle has seen a lot of development over the past two years and five releases: cache relocation, credential rolling, a new auth method, an improved user interface, better verbosity control, and retries. [Read more ...](/blog/2021/07/gargle-1-2-0/) 2021/07/04 [![](/blog/2019/02/bigrquery-1-1-0/bigrquery-1-1-0-sq.jpg)](/blog/2019/02/bigrquery-1-1-0/) [bigrquery 1.1.0](/blog/2019/02/bigrquery-1-1-0/) [package](/categories/package) Mara Averick bigrquery 1.1.0 is now on CRAN. [Read more ...](/blog/2019/02/bigrquery-1-1-0/) 2019/02/15 [![](/blog/2018/04/bigrquery-1-0-0/bigrquery-1-0-0-sq.jpg)](/blog/2018/04/bigrquery-1-0-0/) [bigrquery 1.0.0](/blog/2018/04/bigrquery-1-0-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2018/04/bigrquery-1-0-0/) 2018/04/24 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2 3.5.0: Axes [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) ggplot2 3.5.0: Axes =================== ![](/blog/2024/02/ggplot2-3-5-0-axes/thumbnail-wd.jpg) Photo by [CHUTTERSNAP](https://unsplash.com/photos/white-and-black-measuring-tape-9rSP3SRUYh4)   2024/02/28   [ggplot2](/tags/ggplot2/) , [ggplot2-3-5-0](/tags/ggplot2-3-5-0/)   Teun van den Brand We are pleased to release [ggplot2](https://ggplot2.tidyverse.org) 3.5.0. This release is a large one, so we have split the updates into multiple posts. This posts outlines changes to axes; see the [main release post](/blog/2024/02/ggplot2-3-5-0/) to learn about other changes. Axes, alongside [legends](/blog/2024/02/ggplot2-3-5-0-legends/) , are visual representations of scales and allow observers to read values off of a plot. The innards of axes, like other guides, underwent a major overhaul with the guide system rewrite. Axes specifically are guides for positions and classically display labelled tick marks. In Cartesian coordinates, these are the x- and y-positions, but in non-Cartesian systems may reflect a theta, radius, longitude or latitude. In ggplot2, an axis is usually represented by the [`guide_axis()`](https://ggplot2.tidyverse.org/reference/guide_axis.html) function. We outline the following changes to axes: * [Minor ticks](#minor-ticks) * [Capping the axis line](#capping) * [Logartihmic axes](#logarithmic-axes) * [Stacking axes](#stacked-axes) * [Display in facets](#display-in-facets) Minor ticks[](#minor-ticks) ---------------------------- A much requested expansion of axis capabilities is the ability to draw minor ticks. To draw minor ticks, you can use the `minor.ticks` argument of [`guide_axis()`](https://ggplot2.tidyverse.org/reference/guide_axis.html) . library(ggplot2) p <- ggplot(mpg, aes(displ, hwy)) + geom_point() + guides( x = guide_axis(minor.ticks = TRUE), y = guide_axis(minor.ticks = TRUE) ) p ![Scatterplot of engine displacement versus highway miles per gallon. Both the x and y axes have smaller ticks in between normal ticks.](figs/minor_ticks-1.png) The minor ticks are unlabelled ticks and follow the `minor_breaks` provided to the scale. Their length is determined by the `axis.minor.ticks.length` and their positional children. The rest of their appearance is inherited from the major ticks, as can be seen in the plot below where the minor ticks on the y-axis are also blue. To tweak their style separately from the major ticks, the `axis.minor.ticks.{x.bottom/x.top/y.left/y.right}` setting can be used. Please note that there is _no_ `axis.minor.ticks` setting without the position suffixes, as they inherit from the major ticks. p + scale_x_continuous(minor_breaks = scales::breaks_width(0.2)) + theme( axis.ticks.length = unit(5, "pt"), axis.minor.ticks.length = rel(0.5), axis.minor.ticks.x.bottom = element_line(colour = 'red'), axis.ticks.y = element_line(colour = "blue") ) ![Scatterplot of engine displacement versus highway miles per gallon. The y-axis has blue larger and smaller tick marks, whereas the x-axis has the larger ticks in black and the smaller ticks in red. The x-axis has 4 smaller ticks in between large ones and the smaller ticks are half the size of larger ticks.](figs/minor_ticks_theming-1.png) Capping[](#capping) -------------------- Axes can now also be ‘capped’ at the upper and lower end. We hesitate to call this improvement ‘new’, as it has been a part of base R plotting since time immemorial. When axes are capped, the axis line will not be drawn up to the panel edge, but up to the first and last breaks. Unsurprisingly, this only affects plots where the axis line is not blank. ggplot(mpg, aes(displ, hwy)) + geom_point() + guides( x = guide_axis(cap = "both"), # Cap both ends y = guide_axis(cap = "upper") # Cap the upper end ) + theme(axis.line = element_line()) ![Scatterplot of engine displacement versus highway miles per gallon. The y-axis line starts at the bottom of the panel and continues to the top break. The x-axis line starts at the most left break and ends at the most right break.](figs/capped_axes-1.png) Logarithmic axes[](#logarithmic-axes) -------------------------------------- A new axis for displaying logarithmic (and related) scales has been added: [`guide_axis_logticks()`](https://ggplot2.tidyverse.org/reference/guide_axis_logticks.html) . This axis draws three types of tick marks at log10-spaced positions. The ticks positions are placed in the original, untransformed data-space, so the axis plays well with scale- and coord-transformations. To accommodate a series of logarithmic-like transformations, such as [`scales::transform_pseudo_log()`](https://scales.r-lib.org/reference/transform_log.html) or [`scales::transform_asinh()`](https://scales.r-lib.org/reference/transform_asinh.html) , scales that include 0 in their limits have the ticks mirrored around 0. r <- seq(0.001, 0.999, length.out = 100) df <- data.frame( x = qcauchy(r), y = qlnorm(r) ) p <- ggplot(df, aes(x, y)) + geom_line() + coord_trans(y = "reverse") + scale_y_continuous( transform = "log10", breaks = c(0.1, 1, 10), guide = guide_axis_logticks(long = 2, mid = 1, short = 0.5) ) + scale_x_continuous( transform = "asinh", breaks = c(-100, -10, -1, 0, 1, 10, 100), guide = "axis_logticks" ) p ![A line plot showing a negatively sloped line with a reversed log10-transformation on the y-axis and inverse hyberbolic sine transformation on the x-axis. Large ticks appears at multiples of 10, medium ticks at multiples of 5 and small ticks at multiples of 1.](figs/log_axes-1.png) The log-ticks axis supersedes the earlier [`annotation_logticks()`](https://ggplot2.tidyverse.org/reference/annotation_logticks.html) function. Because it is implemented as an axis, it has minimal fuss with the placement of labels and is immune to the clipping options in the coord. To mirror [`annotation_logticks()`](https://ggplot2.tidyverse.org/reference/annotation_logticks.html) more closely, you can set a negative tick length in the theme. p + theme(axis.ticks.length = unit(-2.25, "pt")) ![The same plot as above, but the tick marks now point inwards.](figs/log_ticks_inward-1.png) Stacked axes[](#stacked-axes) ------------------------------ The last new axis is technically not an axis, but a way to combine axes. [`guide_axis_stack()`](https://ggplot2.tidyverse.org/reference/guide_axis_stack.html) can take multiple other axes and combine them by placing them next to one another. On its own, the usefulness of stacking axes is pretty limited. However, when extensions start defining custom position guides, it is an easy way to mix-and-match axes from different extensions. The first axis is placed next to the panel and subsequent axes are placed further away from the panel. Axes, like legends, have acquired a `theme` argument that can be used to customise the display of individual axes. Currently, there is not a compelling case to use [`guide_axis_stack()`](https://ggplot2.tidyverse.org/reference/guide_axis_stack.html) , but it is an important building block for when axis extensions arrive. Display in facets[](#display-in-facets) ---------------------------------------- More of an indirect improvement to axes, is the ability of facets to tweak the appearance of inner axes when scales are fixed. This facilitates requirements in some journals that every panel should have labelled axes. [`facet_wrap()`](https://ggplot2.tidyverse.org/reference/facet_wrap.html) and [`facet_grid()`](https://ggplot2.tidyverse.org/reference/facet_grid.html) would previously only display axes in between panels when `scales = "free"` was set. This is still the case, but there are more options available for [`facet_grid()`](https://ggplot2.tidyverse.org/reference/facet_grid.html) and fixed scales. Using the `axes = "all"` option, all axes are displayed, including those in between panels. When using `axes = "all_x"` or `axes = "all_y"`, you can narrow down which axes are displayed. p <- ggplot(mpg, aes(displ, hwy)) + geom_point() p + facet_grid(year ~ drv, axes = "all_y") ![A scatterplot facetted by the 'drv' and 'year' variables. The x-axes appear only at the bottom panels, whereas y-axes are displayed for every panel.](figs/facet_axes_display-1.png) In addition, you can choose to selectively suppress labels and only show ticks marks by using the `axis.labels` argument. p + facet_grid(year ~ drv, axes = "all", axis.labels = "all_y") ![A scatterplot facetted by the 'drv' and 'year' variables. The x-axes appear in full only at the bottom panels, and as tick marks in the first row of panels. The y-axes are displayed in full at every panel.](figs/facet_axes_label_display-1.png) That wraps up the visible changes to axes for this post. To read about general changes, see the [main post](/blog/2024/02/ggplot2-3-5-0/) . The changes to legends are covered in a [separate post](/blog/2024/02/ggplot2-3-5-0-legends/) and for the new polar coordinate system (and their axes) will be in a future post. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # withr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Withr [![](/blog/2024/01/withr-3-0-0/thumbnail-sq.jpg)](/blog/2024/01/withr-3-0-0/) [withr 3.0.0](/blog/2024/01/withr-3-0-0/) [package](/categories/package) Lionel Henry withr is the tidyverse solution for automatically cleaning up after yourselves (temporary files, options, etc). This milestone makes withr much faster. [Read more ...](/blog/2024/01/withr-3-0-0/) 2024/01/18 [![](/blog/2017/11/withr-2.1.0/withr-2.1.0-sq.jpg)](/blog/2017/11/withr-2.1.0/) [withr 2.1.0](/blog/2017/11/withr-2.1.0/) [package](/categories/package) Jim Hester withr 2.1.0 is now available on CRAN. [Read more ...](/blog/2017/11/withr-2.1.0/) 2017/11/16 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # bigrquery 1.5.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) bigrquery 1.5.0 =============== ![](/blog/2024/01/bigrquery-1-5-0/thumbnail-wd.jpg) Photo by [Aaron Santelices](https://unsplash.com/photos/gr06IVY2YpM)   2024/01/22   [bigrquery](/tags/bigrquery/)   Hadley Wickham We’re stoked to announce the release of [bigrquery](http://bigrquery.r-dbi.org/) 1.5.0. bigrquery makes it easy to work with data stored in [Google BigQuery](https://developers.google.com/bigquery/) , a hosted database for big data. You can install it from CRAN with: install.packages("bigrquery") This has been the first major update to bigrquery for a while, and is mostly about catching up with innovations elsewhere as well as squashing a bunch of smaller annoyances. library(bigrquery) Here’s a summary of the biggest changes: * bigrquery is now [MIT licensed](https://www.tidyverse.org/blog/2021/12/relicensing-packages/) . * Deprecated functions (i.e. those not starting with `bq_`) have been removed. These have been superseded for a long time and were formally deprecated in bigrquery 1.3.0 (2020). * [`bq_table_download()`](https://bigrquery.r-dbi.org/reference/bq_table_download.html) now returns unknown fields as character vectors. In particular, this means that `BIGNUMERIC` and `JSON` columns are downloaded into R for you to process as you wish. [`bq_table_download()`](https://bigrquery.r-dbi.org/reference/bq_table_download.html) now uses the [clock package](https://clock.r-lib.org) to parse dates, leading to a considerable performance improvement and correct parsing for dates prior to 1970-01-01. * bigquery datasets and tables will now appear in the [RStudio connections pane](https://docs.posit.co/ide/user/ide/guide/data/data-connections.html) when connecting with [`DBI::dbConnect()`](https://dbi.r-dbi.org/reference/dbConnect.html) . * `DBI::dbAppendTable(),` [`DBI::dbCreateTable()`](https://dbi.r-dbi.org/reference/dbCreateTable.html) , and [`DBI::dbExecute()`](https://dbi.r-dbi.org/reference/dbExecute.html) are now supported, and [`DBI::dbGetQuery()`](https://dbi.r-dbi.org/reference/dbGetQuery.html) / [`DBI::dbSendQuery()`](https://dbi.r-dbi.org/reference/dbSendQuery.html) support parameterised queries via the `params` argument. [`DBI::dbReadTable()`](https://dbi.r-dbi.org/reference/dbReadTable.html) , [`DBI::dbWriteTable()`](https://dbi.r-dbi.org/reference/dbWriteTable.html) , [`DBI::dbExistsTable()`](https://dbi.r-dbi.org/reference/dbExistsTable.html) , [`DBI::dbRemoveTable()`](https://dbi.r-dbi.org/reference/dbRemoveTable.html) , and [`DBI::dbListFields()`](https://dbi.r-dbi.org/reference/dbListFields.html) now all work with [`DBI::Id()`](https://dbi.r-dbi.org/reference/Id.html) . * bigrquery now uses 2nd edition of dbplyr interface and is compatible with dbplyr 2.4.0. See the [release notes](https://github.com/r-dbi/bigrquery/releases/tag/v1.5.0) for a full list of changes. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 14 folks who helped make this release happen with questions, comments, and code: [@abalter](https://github.com/abalter) , [@ablack3](https://github.com/ablack3) , [@evanrollinsdrumline](https://github.com/evanrollinsdrumline) , [@hadley](https://github.com/hadley) , [@husseyd](https://github.com/husseyd) , [@jacobmpeters](https://github.com/jacobmpeters) , [@jennybc](https://github.com/jennybc) , [@Kvit](https://github.com/Kvit) , [@meztez](https://github.com/meztez) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@mjbroerman](https://github.com/mjbroerman) , [@ncuriale](https://github.com/ncuriale) , and [@rdavis120](https://github.com/rdavis120) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # roxygen2 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Roxygen2 [![](/blog/2024/01/roxygen2-7-3-0/thumbnail-sq.jpg)](/blog/2024/01/roxygen2-7-3-0/) [roxygen2 7.3.0](/blog/2024/01/roxygen2-7-3-0/) [package](/categories/package) Hadley Wickham This release automatically warns if you forget to export an S3 method, regenerates the `NAMESPACE` before documenting the rest of the package, and does a better job generating aliases for the package documentation topic. [Read more ...](/blog/2024/01/roxygen2-7-3-0/) 2024/01/11 [![](/blog/2022/05/roxygen2-7-2-0/thumbnail-sq.jpg)](/blog/2022/05/roxygen2-7-2-0/) [roxygen2 7.2.0](/blog/2022/05/roxygen2-7-2-0/) [package](/categories/package) Hadley Wickham roxygen2 7.2.0 brings improvements to `NAMESPACE` generation, better multiparameter argument inheritance, and improved warnings. [Read more ...](/blog/2022/05/roxygen2-7-2-0/) 2022/05/13 [![](/blog/2020/03/roxygen2-7-1-0/thumb-sq.jpg)](/blog/2020/03/roxygen2-7-1-0/) [roxygen2 7.1.0](/blog/2020/03/roxygen2-7-1-0/) [package](/categories/package) Gábor Csárdi A minor roxygen2 release with some new features [Read more ...](/blog/2020/03/roxygen2-7-1-0/) 2020/03/11 [![](/blog/2019/11/roxygen2-7-0-0/thumb-sq.jpg)](/blog/2019/11/roxygen2-7-0-0/) [roxygen2 7.0.0](/blog/2019/11/roxygen2-7-0-0/) [package](/categories/package) Hadley Wickham A massive update to roxygen2 now on CRAN. [Read more ...](/blog/2019/11/roxygen2-7-0-0/) 2019/11/12 [![](/blog/2018/08/roxygen2-6-1-0/roxygen2-6-1-0-sq.jpg)](/blog/2018/08/roxygen2-6-1-0/) [roxygen2 6.1.0](/blog/2018/08/roxygen2-6-1-0/) [package](/categories/package) Mara Averick roxygen2 6.1.0 is now on CRAN! [Read more ...](/blog/2018/08/roxygen2-6-1-0/) 2018/08/07 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # r-lib [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) R-Lib [![](/blog/2024/01/withr-3-0-0/thumbnail-sq.jpg)](/blog/2024/01/withr-3-0-0/) [withr 3.0.0](/blog/2024/01/withr-3-0-0/) [package](/categories/package) Lionel Henry withr is the tidyverse solution for automatically cleaning up after yourselves (temporary files, options, etc). This milestone makes withr much faster. [Read more ...](/blog/2024/01/withr-3-0-0/) 2024/01/18 [![](/blog/2022/09/its-about-time/thumbnail-sq.jpg)](/blog/2022/09/its-about-time/) [It’s about time](/blog/2022/09/its-about-time/) [learn](/categories/learn) Mara Averick Davis Vaughan’s talk from rstudio::conf(2022) on clock, an R package that aims to provide comprehensive and safe handling of date-times. [Read more ...](/blog/2022/09/its-about-time/) 2022/09/28 [![](/blog/2021/11/archive-1-1-2/thumbnail-sq.jpg)](/blog/2021/11/archive-1-1-2/) [archive 1.1.2](/blog/2021/11/archive-1-1-2/) [package](/categories/package) Jim Hester archive 1.1.2 is now on CRAN! archive lets you work with file archives, such as ZIP, tar, 7-Zip and RAR and compression formats like gzip, bzip2, XZ and Zstandard. [Read more ...](/blog/2021/11/archive-1-1-2/) 2021/11/04 [![](/blog/2021/03/clock-0-1-0/thumbnail-sq.jpg)](/blog/2021/03/clock-0-1-0/) [Comprehensive Date-Time Handling for R](/blog/2021/03/clock-0-1-0/) [package](/categories/package) Davis Vaughan Introducing, clock, a new package for working with date-times. [Read more ...](/blog/2021/03/clock-0-1-0/) 2021/03/31 [![](/blog/2020/12/usethis-2-0-0/thumbnail-sq.jpg)](/blog/2020/12/usethis-2-0-0/) [usethis 2.0.0](/blog/2020/12/usethis-2-0-0/) [package](/categories/package) Jenny Bryan This is a big release aimed at improving usability, especially around Git and GitHub functionality. [Read more ...](/blog/2020/12/usethis-2-0-0/) 2020/12/10 [![](/blog/2020/04/self-cleaning-test-fixtures/thumbnail-sq.jpg)](/blog/2020/04/self-cleaning-test-fixtures/) [Self-cleaning test fixtures](/blog/2020/04/self-cleaning-test-fixtures/) [programming](/categories/programming) [learn](/categories/learn) Jenny Bryan A wild romp through environments – namely, the environments associated with functions and tests. How to adopt a low-impact lifestyle. [Read more ...](/blog/2020/04/self-cleaning-test-fixtures/) 2020/04/27 [![](/blog/2020/03/roxygen2-7-1-0/thumb-sq.jpg)](/blog/2020/03/roxygen2-7-1-0/) [roxygen2 7.1.0](/blog/2020/03/roxygen2-7-1-0/) [package](/categories/package) Gábor Csárdi A minor roxygen2 release with some new features [Read more ...](/blog/2020/03/roxygen2-7-1-0/) 2020/03/11 [![](/blog/2020/02/glue-strings-and-tidy-eval/glue-strings-and-tidy-eval-sq.jpg)](/blog/2020/02/glue-strings-and-tidy-eval/) [Tidy eval now supports glue strings](/blog/2020/02/glue-strings-and-tidy-eval/) [package](/categories/package) Lionel Henry [Read more ...](/blog/2020/02/glue-strings-and-tidy-eval/) 2020/02/11 [![](/blog/2020/01/vroom-1-1-0/thumbnail-sq.jpg)](/blog/2020/01/vroom-1-1-0/) [vroom 1.1.0](/blog/2020/01/vroom-1-1-0/) [package](/categories/package) Jim Hester vroom 1.1.0 is now on CRAN! [Read more ...](/blog/2020/01/vroom-1-1-0/) 2020/01/15 [![](/blog/2019/11/scales-1-1-0/thumb-sq.jpg)](/blog/2019/11/scales-1-1-0/) [scales 1.1.0](/blog/2019/11/scales-1-1-0/) [package](/categories/package) Hadley Wickham scales 1.1.0 now available on CRAN. It includes a new naming scheme (with `breaks_` and `labels_` prefixes) and greatly improved documentation. [Read more ...](/blog/2019/11/scales-1-1-0/) 2019/11/18 * [««](/tags/r-lib/) * « * [1](/tags/r-lib/) * [2](/tags/r-lib/page/2/) * [3](/tags/r-lib/page/3/) *  …  * [5](/tags/r-lib/page/5/) * [»](/tags/r-lib/page/2/) * [»»](/tags/r-lib/page/5/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # httr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Httr [![](/blog/2023/11/httr2-1-0-0/thumbnail-sq.jpg)](/blog/2023/11/httr2-1-0-0/) [httr2 1.0.0](/blog/2023/11/httr2-1-0-0/) [package](/categories/package) Hadley Wickham httr2 is the successor to httr, providing a pipeable interface to generate HTTP requests and handle the responses. It’s focussed on the needs of an R user wrapping a modern web API, but is flexible enough to handle just about any HTTP related task. [Read more ...](/blog/2023/11/httr2-1-0-0/) 2023/11/14 [![](/blog/2018/12/httr-1-4-0/httr-1-4-0-sq.jpg)](/blog/2018/12/httr-1-4-0/) [httr 1.4.0](/blog/2018/12/httr-1-4-0/) [package](/categories/package) Mara Averick httr 1.4.0 is now on CRAN! [Read more ...](/blog/2018/12/httr-1-4-0/) 2018/12/18 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # package maintenance [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Package Maintenance [![](/blog/2023/11/tidymodels-errors-q4/thumbnail-sq.jpg)](/blog/2023/11/tidymodels-errors-q4/) [Three ways errors are about to get better in tidymodels](/blog/2023/11/tidymodels-errors-q4/) [programming](/categories/programming) Simon Couch The tidymodels team’s biannual spring cleaning gave us a chance to revisit the way we raise some error messages. [Read more ...](/blog/2023/11/tidymodels-errors-q4/) 2023/11/10 [![](/blog/2023/06/spring-cleaning-2023/thumbnail-sq.jpg)](/blog/2023/06/spring-cleaning-2023/) [Package spring cleaning](/blog/2023/06/spring-cleaning-2023/) [other](/categories/other) Andy Teucher When Spring comes around, it’s time to embark on some Spring Cleaning to take care of the package maintenance tasks that you never seem to get around to. This post outlines the processes and tools that the tidyverse team uses to make this job fun and efficient. [Read more ...](/blog/2023/06/spring-cleaning-2023/) 2023/06/07 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # scales [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Scales [![](/blog/2023/11/scales-1-3-0/thumbnail-sq.jpg)](/blog/2023/11/scales-1-3-0/) [scales 1.3.0](/blog/2023/11/scales-1-3-0/) [package](/categories/package) Thomas Lin Pedersen scales 1.3.0 is a minor release focusing on streamlining the API and gradual improvements of the existing utilities [Read more ...](/blog/2023/11/scales-1-3-0/) 2023/11/27 [![](/blog/2022/04/scales-1-2-0/thumbnail-sq.jpg)](/blog/2022/04/scales-1-2-0/) [scales 1.2.0](/blog/2022/04/scales-1-2-0/) [package](/categories/package) Hadley Wickham scales 1.2.0 brings a number of small but useful improvements to numeric labels. [Read more ...](/blog/2022/04/scales-1-2-0/) 2022/04/13 [![](/blog/2019/11/scales-1-1-0/thumb-sq.jpg)](/blog/2019/11/scales-1-1-0/) [scales 1.1.0](/blog/2019/11/scales-1-1-0/) [package](/categories/package) Hadley Wickham scales 1.1.0 now available on CRAN. It includes a new naming scheme (with `breaks_` and `labels_` prefixes) and greatly improved documentation. [Read more ...](/blog/2019/11/scales-1-1-0/) 2019/11/18 [![](/blog/2018/08/scales-1-0-0/scales-1-0-0-sq.jpg)](/blog/2018/08/scales-1-0-0/) [scales 1.0.0](/blog/2018/08/scales-1-0-0/) [package](/categories/package) Dana Seidel scales 1.0.0 is now on CRAN. [Read more ...](/blog/2018/08/scales-1-0-0/) 2018/08/09 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # ggplot2 3.5.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) ggplot2 3.5.0 ============= ![](/blog/2024/02/ggplot2-3-5-0/thumbnail-wd.jpg) Photo by [Megan Bucknall](https://unsplash.com/photos/assorted-color-and-pattern-blanket-lot-dl-Lb5TMxF0)   2024/02/23   [ggplot2](/tags/ggplot2/) , [ggplot2-3-5-0](/tags/ggplot2-3-5-0/)   Teun van den Brand We’re tickled pink to announce the release of [ggplot2](https://ggplot2.tidyverse.org) 3.5.0. ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details. You can install it from CRAN with: install.packages("ggplot2") This blog post will cover a bunch of new features included in the latest release. In addition to rewriting the guide system, we made progress supporting newer R graphics capabilities, re-purposed the use of [`I()`](https://rdrr.io/r/base/AsIs.html) , and introduce an improved polar coordinate system, along with other improvements. As the release is quite large, we are making a [series of blog posts](https://www.tidyverse.org/tags/ggplot2-3-5-0/) covering the major changes. You can see a full list of changes in the [release notes](https://ggplot2.tidyverse.org/news/index.html) library(ggplot2) library(patchwork) library(grid) Guide rewrite[](#guide-rewrite) -------------------------------- Axes and legends, collectively called guides, are an important component to plots, as they allow the translation of visual information back to data qualities. The extension mechanism of ggplot2 allows others to develop their own layers, facets, coords and scales through the ggproto object-oriented system. Finally, after years of being the only major system in ggplot2 still clinging to the S3 system, guides have been rewritten to use ggproto. With this rewrite, guides officially become an extension point that let developers implement their own guides. We have added a section to the [Extending ggplot2](https://ggplot2.tidyverse.org/articles/extending-ggplot2.html#creating-new-guides) vignette on how to develop a new guide. Alongside the rewrite, we made a slew of improvements to guides along the way. As these are somewhat meaty and focused topics, we are going to cover them in separate blog posts about axes and legends. Patterns and gradients[](#patterns-and-gradients) -------------------------------------------------- Patterns and gradients are provided by the grid package, which ggplot2 builds on top of. They were first introduced in R 4.1.0 and were refined in R 4.2.0 to support multiple patterns and gradients. If your graphics device supported it, theme elements could already be set to patterns or gradients, even before this release. > Note: On Windows machines, the default device in RStudio and in the knitr package is [`png()`](https://rdrr.io/r/grDevices/png.html) > , which does not support patterns. In RStudio, you can go to ‘Tools > Global Options > General > Graphics’ and choose the ‘ragg’ or ‘Cairo PNG’ device from the dropdown menu to display patterns. gray_gradient <- linearGradient(scales::pal_grey()(10)) ggplot(mpg, aes(displ, hwy)) + geom_point() + theme(panel.background = element_rect(fill = gray_gradient)) ![Scatterplot of engine displacement versus highway miles per gallon. The panel background is a colour gradient starting from dark grey in the bottom-left corner ending at light grey in the upper-right corner.](figs/theme_gradient-1.png) We are pleased to report that as of this release, patterns can be used as the `fill` aesthetic in most layers. To use a pattern, first build a gradient using {grid}‘s [`linearGradient()`](https://rdrr.io/r/grid/patterns.html) , [`radialGradient()`](https://rdrr.io/r/grid/patterns.html) functions, or a pattern using the [`pattern()`](https://rdrr.io/r/grid/patterns.html) function. Because handling patterns and gradients is very similar, we will treat gradients as if they were patterns: when we say ‘pattern’ in the text below, please mind that we mean patterns and gradients alike. These patterns can be passed to a layer as the `fill` aesthetic. Below, you can see two behaviours of the [`linearGradient()`](https://rdrr.io/r/grid/patterns.html) pattern, depending on its `group` argument. The pattern with `group = FALSE` will display the gradient in every rectangle and `group = TRUE` will apply the gradient to all rectangles together. colours <- scales::viridis_pal()(10) grad_ungroup <- linearGradient(colours, group = FALSE) grad_grouped <- linearGradient(colours, group = TRUE) ungroup <- ggplot(mpg, aes(factor(cyl))) + geom_bar(fill = grad_ungroup) + labs(title = "Ungrouped gradient") grouped <- ggplot(mpg, aes(factor(cyl))) + geom_bar(fill = grad_grouped) + labs(title = "Grouped gradient") ungroup | grouped ![Two barplots showing the counts of number of cylinders. The first plot is titled 'Ungrouped gradient' and shows individual gradients in the bars. The second is titled 'Grouped gradient' and shows a single gradient along all bars.](figs/grouping_gradient-1.png) Besides passing a static pattern as the `fill` aesthetic, it is also possible to map values to patterns using [`scale_fill_manual()`](https://ggplot2.tidyverse.org/reference/scale_manual.html) . To map values to patterns, pass a list of patterns to the `values` argument of the scale. When providing patterns as a list, the list can be a mix of patterns and plain colours, like `"limegreen"` in the plot below. We are excited that people may come up with nice pattern palettes that can be used in similar fashion. patterns <- list( linearGradient(colours, group = FALSE), "limegreen", radialGradient(colours, group = FALSE), pattern( rectGrob(x = c(0.25, 0.75), y = c(0.25, 0.75), width = 0.5, height = 0.5), width = unit(5, "mm"), height = unit(5, "mm"), extend = "repeat", gp = gpar(fill = "limegreen") ) ) ggplot(mpg, aes(factor(cyl), fill = factor(cyl))) + geom_bar() + scale_fill_manual(values = patterns) ![Barplot showing counts of number of cylinders with the bars filled by a linear gradient, a plain green colour, a radial gradient and a green checkerboard pattern.](figs/pattern_scale-1.png) The largest obstacle we had to overcome to support gradients in ggplot2 was to apply the `alpha` aesthetic consistently to the patterns. The regular [`scales::alpha()`](https://scales.r-lib.org/reference/alpha.html) function does not work with patterns, so we implemented a new [`fill_alpha()`](https://ggplot2.tidyverse.org/reference/fill_alpha.html) function that applies the `alpha` aesthetic to the patterns. By switching out `fill = alpha(fill, alpha)` with `fill = fill_alpha(fill, alpha)` in the [`grid::gpar()`](https://rdrr.io/r/grid/gpar.html) function, extension developers can enable pattern fills in their own layer extensions. The [`fill_alpha()`](https://ggplot2.tidyverse.org/reference/fill_alpha.html) function checks if the active device supports patterns and spits out a friendlier warning or error on demand. For extension developers that want to use newer graphics features, you can reuse the [`check_device()`](https://ggplot2.tidyverse.org/reference/check_device.html) function to check feature availability or throw messages in a similar fashion. # The currently active device is the ragg::agg_png() device check_device(feature = "patterns", action = "test") #> [1] TRUE check_device(feature = "glyphs", action = "abort") #> Error: #> ! The agg_png device does not support typeset glyphs. Ignoring scales[](#ignoring-scales) ------------------------------------ In this release, ggplot2 has changed how the plots interact with variables created with [`I()`](https://rdrr.io/r/base/AsIs.html) (‘AsIs’ variables). The change is somewhat subtle, so it takes a bit of explaining. It _used to be_ the case that ‘AsIs’ variables automatically added an identity scale to the plot. Identity scales in ggplot2 preserve the original input, without mapping or transforming them. For example, iif you give literal colour names as the `colour` aesthetic, the plot will use these exact colours. set.seed(42) my_colours <- sample(c("red", "green", "blue"), nrow(mpg), replace = TRUE) ggplot(mpg, aes(displ, hwy)) + geom_point(aes(colour = my_colours)) + scale_colour_identity() ![Scatterplot of engine displacement versus highway miles per gallon with points in red, green and blue.](figs/literal_colours-1.png) However, because identity scales _are_ true scales, you cannot combine literal colours in one layer with mapped colours in the next. Trying to do so, will confront you with the ‘unknown colour name’ error. ggplot(mpg, aes(displ, hwy)) + geom_point(aes(colour = drv), shape = 1, size = 5) + geom_point(aes(colour = my_colours)) + scale_colour_identity() #> Error in `geom_point()`: #> ! Problem while converting geom to grob. #> ℹ Error occurred in the 1st layer. #> Caused by error: #> ! Unknown colour name: f In order to prevent such clashes between identity scales that map nothing and regular scales, we have changed how ‘AsIs’ variables interact with scales. Instead of adding an identity scale, ‘AsIs’ variables are now altogether _ignored_ by the scale systems. On the surface, the new behaviour is very similar to the old one, in that for example literal colours are used. However, with ‘AsIs’ variables ignored, you can now freely combine layers with ‘AsIs’ input with layers that map input. If you need a legend for the literal variable, we recommend to use the identity scale mechanism instead. ggplot(mpg, aes(displ, hwy)) + geom_point(aes(colour = drv), shape = 1, size = 5) + geom_point(aes(colour = I(my_colours)), show.legend = FALSE) ![Scatterplot of engine displacement versus highway miles per gallon. Every point has two circles: a smaller one in red, green or blue and a larger one mapped to the 'drv' variable.](figs/asis_aesthetic-1.png) Perhaps more salient than avoid scale clashes, is that the same applies to the `x` and `y` position aesthetics. There has never been a `scale_x_identity()` or `scale_y_identity()` function, so what this means may be unexpected. Internally, scales transform every continuous variable to the 0-1 range before drawing the graphics. So too do ‘AsIs’ position aesthetics work: you can use numbers between 0 and 1 to set the position. These positions are relative to the plot’s panel and this mechanism opens up a great way to add plot annotations that are independent of the data. t <- seq(0, 2 * pi, length.out = 100) ggplot(mpg, aes(displ, hwy)) + geom_point(colour = "grey50") + annotate( "rect", xmin = I(0.05), xmax = I(0.95), ymin = I(0.05), ymax = I(0.95), fill = NA, colour = "red" ) + annotate( "path", x = I(cos(t) / 2 + 0.5), y = I(sin(t) / 2 + 0.5), colour = "blue" ) + annotate( "text", label = "Text in the middle", x = I(0.5), y = I(0.5), size = 8 ) ![Scatterplot of engine displacement versus highway miles per gallon. The plot has a red rectangle slightly smaller than the panel, a blue circle touching the panel edges and text in the middle that reads: 'text in the middle'.](figs/asis_annotation-1.png) Please take note that discrete variables as ‘AsIs’ position aesthetic have no interpretation and will likely result in errors. Other improvements[](#other-improvements) ------------------------------------------ Coordinating text sizes between the theme and [`geom_text()`](https://ggplot2.tidyverse.org/reference/geom_text.html) / [`geom_label()`](https://ggplot2.tidyverse.org/reference/geom_text.html) has been a hassle, since the theme uses text sizes in points (pt) and geoms use text size in millimetres. Now, one can control what the `size` aesthetic means for text, by setting the `size.unit` argument. p <- ggplot(mtcars, aes(wt, mpg, label = rownames(mtcars))) p + geom_text(size = 10, size.unit = "pt") + theme(axis.text = element_text(size = 10)) ![A plot showing weight versus miles per gallon with individual cars labelled by text. The text in the plot has the same size as the text labelling the axes.](figs/size_unit_arg-1.png) Two improvements have been made to [`geom_label()`](https://ggplot2.tidyverse.org/reference/geom_text.html) . The first is that it now obeys an `angle` aesthetic. p + geom_label(aes(angle = runif(nrow(mtcars), -45, 45))) ![A plot showing weight versus miles per gallon with individual cars labelled by textboxes. The textboxes are displayed in different angles.](figs/label_angle-1.png) In addition, [`geom_label()`](https://ggplot2.tidyverse.org/reference/geom_text.html) ‘s `label.padding` argument can be controlled individually for every side of the text by using the [`margin()`](https://ggplot2.tidyverse.org/reference/element.html) function. The legend keys for labels has also changed to reflect the geom more accurately. p + geom_label( aes(colour = factor(cyl)), label.padding = margin(t = 2, r = 20, b = 1, l = 0) ) ![A plot showing weight versus miles per gallon with individual cars labelled by textboxes. The textboxes have a large margin on the right.](figs/label_padding-1.png) Like [`geom_density()`](https://ggplot2.tidyverse.org/reference/geom_density.html) before it, [`geom_violin()`](https://ggplot2.tidyverse.org/reference/geom_violin.html) now gains a `bounds` argument to restrict the range wherein density is estimated. df <- data.frame( x = c(rbeta(100, 0.5, 0.5), rbeta(100, 1, 1), rbeta(100, 2, 2)), group = rep(c("A", "B", "C"), each = 100) ) ggplot(df, aes(group, x)) + geom_violin(bounds = c(0, 1)) ![Violin plot showing random numbers drawn from beta distributions with different parameters. The ends of the first two violins are flat at the top and bottom.](figs/violin_bounds-1.png) The [`geom_boxplot()`](https://ggplot2.tidyverse.org/reference/geom_boxplot.html) has acquired an option to remove (rather than hide) outliers. Setting `outliers = FALSE` removes outliers so that the plot limits do not take these into account. For hiding (and not removing) outliers, you can still set `outlier.shape = NA`. Also, it has gained a `staplewidth` argument that can be used to draw staples: horizontal lines at the end of the boxplot whiskers. The default, `staplewidth = 0`, will suppress the staples so your current box plots continue to look the same. ggplot(diamonds, aes(cut, price)) + geom_boxplot(outliers = FALSE, staplewidth = 0.5) ![Boxplot showing the price of diamonds per cut. The y-axis does not go much beyond the whiskers, and whiskers are decorated with a staple.](figs/boxplot_outliers_staples-1.png) The scales functions now do a better job at reporting _which_ scale has encountered an error. scale_colour_brewer(breaks = 1:5, labels = 1:4) #> Error in `scale_colour_brewer()`: #> ! `breaks` and `labels` must have the same length. ggplot(mpg, aes(class, displ)) + geom_boxplot() + scale_x_continuous() #> Error in `scale_x_continuous()`: #> ! Discrete values supplied to continuous scale. #> ℹ Example values: "compact", "compact", "compact", "compact", and "compact" ggplot(msleep, aes(bodywt - 1, brainwt)) + geom_point(na.rm = TRUE) + scale_x_log10() #> Warning in transformation$transform(x): NaNs produced #> Warning in scale_x_log10(): log-10 transformation introduced infinite values. ![Scatterplot showing body weight minus one versus brain weight of mammals. The x-axis is log-transformed.](figs/scale_messages-1.png) Acknowledgements[](#acknowledgements) -------------------------------------- Thank you to all people who have contributed issues, code and comments to this release: [@92amartins](https://github.com/92amartins) , [@a-torgovitsky](https://github.com/a-torgovitsky) , [@aarongraybill](https://github.com/aarongraybill) , [@aavogt](https://github.com/aavogt) , [@agila5](https://github.com/agila5) , [@ahcyip](https://github.com/ahcyip) , [@AlexanderCasper](https://github.com/AlexanderCasper) , [@alexkrohn](https://github.com/alexkrohn) , [@alofting](https://github.com/alofting) , [@andrewgustar](https://github.com/andrewgustar) , [@antagomir](https://github.com/antagomir) , [@aphalo](https://github.com/aphalo) , [@Ari04T](https://github.com/Ari04T) , [@AroneyS](https://github.com/AroneyS) , [@Asa12138](https://github.com/Asa12138) , [@ashgreat](https://github.com/ashgreat) , [@averissimo](https://github.com/averissimo) , [@bakerwm](https://github.com/bakerwm) , [@balling-dev](https://github.com/balling-dev) , [@banbh](https://github.com/banbh) , [@barracuda156](https://github.com/barracuda156) , [@BartJanvanRossum](https://github.com/BartJanvanRossum) , [@beansrowning](https://github.com/beansrowning) , [@benimwolfspelz](https://github.com/benimwolfspelz) , [@bfordAIMS](https://github.com/bfordAIMS) , [@bguiastr](https://github.com/bguiastr) , [@bnicenboim](https://github.com/bnicenboim) , [@BrianDiggs](https://github.com/BrianDiggs) , [@bsgerber](https://github.com/bsgerber) , [@burrapreeti](https://github.com/burrapreeti) , [@bwiernik](https://github.com/bwiernik) , [@ccsarapas](https://github.com/ccsarapas) , [@CGlemser](https://github.com/CGlemser) , [@chiajungTung](https://github.com/chiajungTung) , [@chipsin87](https://github.com/chipsin87) , [@cjvanlissa](https://github.com/cjvanlissa) , [@CorradoLanera](https://github.com/CorradoLanera) , [@danielneilson](https://github.com/danielneilson) , [@danli349](https://github.com/danli349) , [@DasHammett](https://github.com/DasHammett) , [@davidhodge931](https://github.com/davidhodge931) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dieghernan](https://github.com/dieghernan) , [@Ductmonkey](https://github.com/Ductmonkey) , [@edent](https://github.com/edent) , [@Elham-adabi](https://github.com/Elham-adabi) , [@ELICHOS](https://github.com/ELICHOS) , [@eliocamp](https://github.com/eliocamp) , [@ellisp](https://github.com/ellisp) , [@emuise](https://github.com/emuise) , [@erikdeluca](https://github.com/erikdeluca) , [@f2il-kieranmace](https://github.com/f2il-kieranmace) , [@FDylanT](https://github.com/FDylanT) , [@fkohrt](https://github.com/fkohrt) , [@francisbarton](https://github.com/francisbarton) , [@fredcallaway](https://github.com/fredcallaway) , [@frezza-metabolomics](https://github.com/frezza-metabolomics) , [@GabrielHoffman](https://github.com/GabrielHoffman) , [@gaospecial](https://github.com/gaospecial) , [@garyzhubc](https://github.com/garyzhubc) , [@gavinsimpson](https://github.com/gavinsimpson) , [@Generalized](https://github.com/Generalized) , [@ghost](https://github.com/ghost) , [@giadasp](https://github.com/giadasp) , [@GMSL1](https://github.com/GMSL1) , [@grantmcdermott](https://github.com/grantmcdermott) , [@hadley](https://github.com/hadley) , [@hlynurhallgrims](https://github.com/hlynurhallgrims) , [@holgerbrandl](https://github.com/holgerbrandl) , [@hpages](https://github.com/hpages) , [@HRodenhizer](https://github.com/HRodenhizer) , [@hub-shale](https://github.com/hub-shale) , [@hughjonesd](https://github.com/hughjonesd) , [@ibuiltthis](https://github.com/ibuiltthis) , [@ingewortel](https://github.com/ingewortel) , [@isaacvock](https://github.com/isaacvock) , [@Istalan](https://github.com/Istalan) , [@istvankleijn](https://github.com/istvankleijn) , [@jacobkasper](https://github.com/jacobkasper) , [@jammainen](https://github.com/jammainen) , [@jan-glx](https://github.com/jan-glx) , [@JaredAllen2](https://github.com/JaredAllen2) , [@jashapiro](https://github.com/jashapiro) , [@jimjam-slam](https://github.com/jimjam-slam) , [@jmuhlenkamp](https://github.com/jmuhlenkamp) , [@jonspring](https://github.com/jonspring) , [@JorisChau](https://github.com/JorisChau) , [@joshhwuu](https://github.com/joshhwuu) , [@jpeasari](https://github.com/jpeasari) , [@jromanowska](https://github.com/jromanowska) , [@jsacerot](https://github.com/jsacerot) , [@jtlandis](https://github.com/jtlandis) , [@jtr13](https://github.com/jtr13) , [@jttoivon](https://github.com/jttoivon) , [@karchern](https://github.com/karchern) , [@klin333](https://github.com/klin333) , [@kmavrommatis](https://github.com/kmavrommatis) , [@kramerrs](https://github.com/kramerrs) , [@krlmlr](https://github.com/krlmlr) , [@kylebutts](https://github.com/kylebutts) , [@larmarange](https://github.com/larmarange) , [@latot](https://github.com/latot) , [@lhami](https://github.com/lhami) , [@liang09255](https://github.com/liang09255) , [@linzi-sg](https://github.com/linzi-sg) , [@lionel-](https://github.com/lionel-) , [@lnarwhale](https://github.com/lnarwhale) , [@manjumc1975](https://github.com/manjumc1975) , [@mariadelmarq](https://github.com/mariadelmarq) , [@matanhakim](https://github.com/matanhakim) , [@math-mcshane](https://github.com/math-mcshane) , [@mattgalbraith](https://github.com/mattgalbraith) , [@matthewjnield](https://github.com/matthewjnield) , [@mcwayrm](https://github.com/mcwayrm) , [@melissagwolf](https://github.com/melissagwolf) , [@MichaelChirico](https://github.com/MichaelChirico) , [@MikkoVihtakari](https://github.com/MikkoVihtakari) , [@MjelleLab](https://github.com/MjelleLab) , [@mjskay](https://github.com/mjskay) , [@mkoohafkan](https://github.com/mkoohafkan) , [@mmokrejs](https://github.com/mmokrejs) , [@modmost](https://github.com/modmost) , [@moodymudskipper](https://github.com/moodymudskipper) , [@morrisseyj](https://github.com/morrisseyj) , [@mps9506](https://github.com/mps9506) , [@Nh-code](https://github.com/Nh-code) , [@njtierney](https://github.com/njtierney) , [@oliviercailloux](https://github.com/oliviercailloux) , [@olivroy](https://github.com/olivroy) , [@otaviolovison](https://github.com/otaviolovison) , [@pablobernabeu](https://github.com/pablobernabeu) , [@paulatn240](https://github.com/paulatn240) , [@phauchamps](https://github.com/phauchamps) , [@quantixed](https://github.com/quantixed) , [@ralmond](https://github.com/ralmond) , [@ramiromagno](https://github.com/ramiromagno) , [@reallzg](https://github.com/reallzg) , [@retodomax](https://github.com/retodomax) , [@robbiebatley](https://github.com/robbiebatley) , [@Rong-Zh](https://github.com/Rong-Zh) , [@rossellhayes](https://github.com/rossellhayes) , [@RoyalTS](https://github.com/RoyalTS) , [@rvalieris](https://github.com/rvalieris) , [@s-andrews](https://github.com/s-andrews) , [@s-elsheikh](https://github.com/s-elsheikh) , [@schloerke](https://github.com/schloerke) , [@Sckende](https://github.com/Sckende) , [@sdmason](https://github.com/sdmason) , [@sirallen](https://github.com/sirallen) , [@slowkow](https://github.com/slowkow) , [@spaette](https://github.com/spaette) , [@steveharoz](https://github.com/steveharoz) , [@sunroofgod](https://github.com/sunroofgod) , [@szimmer](https://github.com/szimmer) , [@tbates](https://github.com/tbates) , [@teunbrand](https://github.com/teunbrand) , [@tfjaeger](https://github.com/tfjaeger) , [@thomasp85](https://github.com/thomasp85) , [@TimBMK](https://github.com/TimBMK) , [@TimTaylor](https://github.com/TimTaylor) , [@tjebo](https://github.com/tjebo) , [@trekonom](https://github.com/trekonom) , [@tungttnguyen](https://github.com/tungttnguyen) , [@twest820](https://github.com/twest820) , [@UliSchopp](https://github.com/UliSchopp) , [@vnijs](https://github.com/vnijs) , [@warnes](https://github.com/warnes) , [@wbvguo](https://github.com/wbvguo) , [@willgearty](https://github.com/willgearty) , [@Yann-C-INN](https://github.com/Yann-C-INN) , [@yannk-lm](https://github.com/yannk-lm) , [@Yunuuuu](https://github.com/Yunuuuu) , [@yutannihilation](https://github.com/yutannihilation) , [@yuw444](https://github.com/yuw444) , [@zekiakyol](https://github.com/zekiakyol) , and [@zhenglukai](https://github.com/zhenglukai) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # roxygen2 7.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) roxygen2 7.3.0 ============== ![](/blog/2024/01/roxygen2-7-3-0/thumbnail-wd.jpg) Photo by [Adi Goldstein](https://unsplash.com/photos/Hli3R6LKibo)   2024/01/11   [roxygen2](/tags/roxygen2/)   Hadley Wickham We’re well pleased to announce the release of [roxygen2](http://roxygen2.r-lib.org/) 7.3.0. roxygen2 allows you to write specially formatted R comments that generate R documentation files (`man/*.Rd`) and the `NAMESPACE` file. roxygen2 is used by over 13,000 CRAN packages. You can install it from CRAN with: install.packages("roxygen2") There are four major improvements in this release: * The `NAMESPACE` roclet now reports if you have S3 methods that are missing an `@export` tag. All S3 methods need to be `@export`ed even if the generic is not. This avoids rare, but hard to debug, problems. If you think this is giving a false positive, [please file an issue](https://github.com/r-lib/roxygen2/issues/new) and suppress the warning with `@exportS3Method NULL`. I’ve also considerably revamped the documentation for S3 methods in [`vignette("namespace")`](https://roxygen2.r-lib.org/dev/articles/namespace.html#s3) . The docs now discuss what exporting an S3 method really means, and why it would be technically better to call it _registering_ the method. * Finally, the `NAMESPACE` roclet once again regenerates imports _before_ loading package code and parsing roxygen blocks. This has been the goal for a [long time](https://github.com/r-lib/roxygen2/issues/372) , but we accidentally broke it when adding support for code execution in markdown blocks. This change resolves a family of problems where you somehow bork your `NAMESPACE` and can’t easily get fix it because you can’t re-document the package because you can’t load your package because your `NAMESPACE` is borked. * `@docType package` now works like [`"_PACKAGE"`](https://roxygen2.r-lib.org/articles/rd-other.html#packages) , including creating a `{packagename}-package` alias automatically. This resolves a bug introduced in roxygen2 7.0.0 that meant that many packages lacked the correct alias for their package documentation topic. * `"_PACKAGE"` does a better job of automatically generating aliases. In particular, it will no longer generate a duplicate alias if you have a function with the same name as your package (like [`glue::glue()`](https://glue.tidyverse.org/reference/glue.html) or [`reprex::reprex()`](https://reprex.tidyverse.org/reference/reprex.html) ). If you’ve previously had to hack around this bug, you can now delete any custom `@aliases` tags associated with the `"_PACKAGE"` docs. You can see a full list of other minor improvements and bug fixes in the [release notes](https://github.com/r-lib/roxygen2/releases/tag/v7.3.0) . Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to the 46 folks who helped make this release possible through their thoughtful questions and carefully crafted code! [@andrewmarx](https://github.com/andrewmarx) , [@ashbythorpe](https://github.com/ashbythorpe) , [@ateucher](https://github.com/ateucher) , [@bahadzie](https://github.com/bahadzie) , [@bastistician](https://github.com/bastistician) , [@beginb](https://github.com/beginb) , [@brodieG](https://github.com/brodieG) , [@bryanhanson](https://github.com/bryanhanson) , [@cbielow](https://github.com/cbielow) , [@daattali](https://github.com/daattali) , [@DanChaltiel](https://github.com/DanChaltiel) , [@dpprdan](https://github.com/dpprdan) , [@dsweber2](https://github.com/dsweber2) , [@espinielli](https://github.com/espinielli) , [@hadley](https://github.com/hadley) , [@hughjonesd](https://github.com/hughjonesd) , [@jeroen](https://github.com/jeroen) , [@jmbarbone](https://github.com/jmbarbone) , [@johnbaums](https://github.com/johnbaums) , [@jonocarroll](https://github.com/jonocarroll) , [@kathi-munk](https://github.com/kathi-munk) , [@krlmlr](https://github.com/krlmlr) , [@kylebutts](https://github.com/kylebutts) , [@lionel-](https://github.com/lionel-) , [@LouisLeNezet](https://github.com/LouisLeNezet) , [@maelle](https://github.com/maelle) , [@MaximilianPi](https://github.com/MaximilianPi) , [@MichaelChirico](https://github.com/MichaelChirico) , [@moodymudskipper](https://github.com/moodymudskipper) , [@msberends](https://github.com/msberends) , [@multimeric](https://github.com/multimeric) , [@musvaage](https://github.com/musvaage) , [@neshvig10](https://github.com/neshvig10) , [@olivroy](https://github.com/olivroy) , [@ralmond](https://github.com/ralmond) , [@RMHogervorst](https://github.com/RMHogervorst) , [@Robinlovelace](https://github.com/Robinlovelace) , [@rossellhayes](https://github.com/rossellhayes) , [@rsbivand](https://github.com/rsbivand) , [@sbgraves237](https://github.com/sbgraves237) , [@schradj](https://github.com/schradj) , [@sebffischer](https://github.com/sebffischer) , [@simonpcouch](https://github.com/simonpcouch) , [@stemangiola](https://github.com/stemangiola) , [@tau31](https://github.com/tau31) , and [@trusch139](https://github.com/trusch139) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Three ways errors are about to get better in tidymodels [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Three ways errors are about to get better in tidymodels ======================================================= ![](/blog/2023/11/tidymodels-errors-q4/thumbnail-wd.jpg) Photo by [Nagesh Badu](https://unsplash.com/photos/vYcH7pI6v1Q)   2023/11/10   [tidymodels](/tags/tidymodels/) , [parsnip](/tags/parsnip/) , [tune](/tags/tune/) , [package maintenance](/tags/package-maintenance/)   Simon Couch Twice a year, the tidymodels team comes together for “spring cleaning,” a week-long project devoted to package maintenance. Ahead of the week, we come up with a list of maintenance tasks that we’d like to see consistently implemented across our packages. Many of these tasks can be completed by running one usethis function, while others are much more involved, like issue triage.[1](#fn:1) In tidymodels, triaging issues in our core packages helps us to better understand common ways that users struggle to wrap their heads around an API choice we’ve made or find the information they need. So, among other things, refinements to the wording of our error messages is a common output of our spring cleanings. This blog post will call out three kinds of changes to our erroring that came out of this spring cleaning: * Improving existing errors: [The outcome went missing](#outcome) * Do something where we once did nothing: [Predicting with things that can’t predict](#predict) * Make a place and point to it: [Model formulas](#model) To demonstrate, we’ll walk through some examples using the tidymodels packages: library(tidymodels) #> ── Attaching packages ──────────────────────────── tidymodels 1.1.1 ── #> ✔ broom 1.0.5 ✔ recipes 1.0.8.9000 #> ✔ dials 1.2.0 ✔ rsample 1.2.0 #> ✔ dplyr 1.1.3 ✔ tibble 3.2.1 #> ✔ ggplot2 3.4.4 ✔ tidyr 1.3.0 #> ✔ infer 1.0.5 ✔ tune 1.1.2.9000 #> ✔ modeldata 1.2.0 ✔ workflows 1.1.3 #> ✔ parsnip 1.1.1.9001 ✔ workflowsets 1.0.1 #> ✔ purrr 1.0.2 ✔ yardstick 1.2.0 #> ── Conflicts ─────────────────────────────── tidymodels_conflicts() ── #> ✖ purrr::discard() masks scales::discard() #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ✖ recipes::step() masks stats::step() #> • Use suppressPackageStartupMessages() to eliminate package startup messages Note that my installed versions include the current dev version of a few tidymodels packages. You can install those versions with: pak::pak(paste0("tidymodels/", c("tune", "parsnip", "recipes"))) The outcome went missing 👻[](#the-outcome-went-missing-) ---------------------------------------------------------- The tidymodels packages focus on _supervised_ machine learning problems, predicting the value of an outcome using predictors.[2](#fn:2) For example, in the code: linear_spec <- linear_reg() linear_fit <- fit(linear_spec, mpg ~ hp, mtcars) The `mpg` variable is the outcome. There are many ways that an analyst may mistakenly fail to pass an outcome. In the most straightforward case, they might omit the outcome on the LHS of the formula: fit(linear_spec, ~ hp, mtcars) #> Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) : #> incompatible dimensions In this case, parsnip used to defer to the modeling engine to raise an error, which may or may not be informative. There are many less obvious ways an analyst may mistakenly supply no outcome variable. For example, try spotting the issue in the following code, defining a recipe to perform principal component analysis (PCA) on the numeric variables in the data before fitting the model: mtcars_rec <- recipe(mpg ~ ., mtcars) %>% step_pca(all_numeric()) workflow(mtcars_rec, linear_spec) %>% fit(mtcars) #> Error: object '.' not found A head-scratcher! To help diagnose what’s happening here, we could first try seeing what data is actually being passed to the model. mtcars_rec_trained <- mtcars_rec %>% prep(mtcars) mtcars_rec_trained %>% bake(NULL) #> # A tibble: 32 × 5 #> PC1 PC2 PC3 PC4 PC5 #> #> 1 -195. 12.8 -11.4 0.0164 2.17 #> 2 -195. 12.9 -11.7 -0.479 2.11 #> 3 -142. 25.9 -16.0 -1.34 -1.18 #> 4 -279. -38.3 -14.0 0.157 -0.817 #> 5 -399. -37.3 -1.38 2.56 -0.444 #> 6 -248. -25.6 -12.2 -3.01 -1.08 #> 7 -435. 20.9 13.9 0.801 -0.916 #> 8 -160. -20.0 -23.3 -1.06 0.787 #> 9 -172. 10.8 -18.3 -4.40 -0.836 #> 10 -209. 19.7 -8.94 -2.58 1.33 #> # ℹ 22 more rows Mmm. What happened to `mpg`? We mistakenly told `step_pca()` to perform PCA on _all_ of the numeric variables, not just the numeric _predictors_! As a result, it incorporated `mpg` into the principal components, removing each of the original numeric variables after the fact. Rewriting using the correct tidyselect specification `all_numeric_predictors()`: mtcars_rec_new <- recipe(mpg ~ ., mtcars) %>% step_pca(all_numeric_predictors()) workflow(mtcars_rec_new, linear_spec) %>% fit(mtcars) #> ══ Workflow [trained] ════════════════════════════════════════════════ #> Preprocessor: Recipe #> Model: linear_reg() #> #> ── Preprocessor ────────────────────────────────────────────────────── #> 1 Recipe Step #> #> • step_pca() #> #> ── Model ───────────────────────────────────────────────────────────── #> #> Call: #> stats::lm(formula = ..y ~ ., data = data) #> #> Coefficients: #> (Intercept) PC1 PC2 PC3 PC4 #> 43.39293 0.07609 -0.05266 0.57892 0.94890 #> PC5 #> -1.72569 Works like a charm. That error we saw previously could be much more helpful, though. With the current developmental version of parsnip, this looks like: fit(linear_spec, ~ hp, mtcars) #> Error: #> ! `linear_reg()` was unable to find an outcome. #> ℹ Ensure that you have specified an outcome column and that it hasn't #> been removed in pre-processing. Or, with workflows: workflow(mtcars_rec, linear_spec) %>% fit(mtcars) #> Error: #> ! `linear_reg()` was unable to find an outcome. #> ℹ Ensure that you have specified an outcome column and that it hasn't #> been removed in pre-processing. Much better. Predicting with things that can’t predict[](#predicting-with-things-that-cant-predict) --------------------------------------------------------------------------------------- Earlier this year, Dr. Louise E. Sinks put out a [wonderful blog post](https://lsinks.github.io/posts/2023-04-10-tidymodels/tidymodels_tutorial.html) documenting what it felt like to approach the various object types defined in the tidymodels as a newcomer to the collection of packages. They wrote: > I found it confusing that `fit`, `last_fit`, `fit_resamples`, etc., did not all produce objects that contained the same information and could be acted on by the same functions. This makes sense. While we try to forefront the intended mental model for fitting and predicting with tidymodels in our APIs and documentation, we also need to be proactive in anticipating common challenges in constructing that mental model. For example, we’ve found that it’s sometimes not clear to users which outputs they can call [`predict()`](https://rdrr.io/r/stats/predict.html) on. One such situation, as Louise points out, is with `fit_resamples()`: # fit a linear regression model to bootstrap resamples of mtcars mtcars_res <- fit_resamples(linear_reg(), mpg ~ ., bootstraps(mtcars)) mtcars_res #> # Resampling results #> # Bootstrap sampling #> # A tibble: 25 × 4 #> splits id .metrics .notes #> #> 1 Bootstrap01 #> 2 Bootstrap02 #> 3 Bootstrap03 #> 4 Bootstrap04 #> 5 Bootstrap05 #> 6 Bootstrap06 #> 7 Bootstrap07 #> 8 Bootstrap08 #> 9 Bootstrap09 #> 10 Bootstrap10 #> # ℹ 15 more rows With previous tidymodels versions, mistakenly trying to predict with this object resulted in the following output: predict(mtcars_res) #> Error in UseMethod("predict") : #> no applicable method for 'predict' applied to an object of class #> "c('resample_results', 'tune_results', 'tbl_df', 'tbl', 'data.frame')" Some R developers may recognize this error as what results when we didn’t define any [`predict()`](https://rdrr.io/r/stats/predict.html) method for `tune_results` objects. We didn’t do so because prediction isn’t well-defined for tuning results. _But_, this error message does little to help a user understand why that’s the case. We’ve recently made some changes to error more informatively in this case. We do so by defining a “dummy” [`predict()`](https://rdrr.io/r/stats/predict.html) method for tuning results, implemented only for the sake of erroring more informatively. The same code will now give the following output: predict(mtcars_res) #> Error in `predict()`: #> ! `predict()` is not well-defined for tuning results. #> ℹ To predict with the optimal model configuration from tuning #> results, ensure that the tuning result was generated with the #> control option `save_workflow = TRUE`, run `fit_best()`, and #> then predict using `predict()` on its output. #> ℹ To collect predictions from tuning results, ensure that the #> tuning result was generated with the control option `save_pred #> = TRUE` and run `collect_predictions()`. References to important concepts or functions, like [control options](https://tune.tidymodels.org/reference/control_grid.html) , [`fit_best()`](https://tune.tidymodels.org/reference/fit_best.html?q=fit_best) , and [`collect_predictions()`](https://tune.tidymodels.org/reference/collect_predictions.html?q=collect) , link to the help-files for those functions using [cli’s erroring tools](https://cli.r-lib.org/reference/cli_abort.html) . We hope new error messages like this will help to get folks back on track. Model formulas[](#model-formulas) ---------------------------------- In R, formulas provide a compact, symbolic notation to specify model terms. Many modeling functions in R make use of “specials,” or nonstandard notations used in formulas. Specials are defined and handled as a special case by a given modeling package. parsnip defers to engine packages to handle specials, so you can work with them as usual. For example, the mgcv package provides support for generalized additive models in R, and defines a special called `s()` to indicate smoothing terms. You can interface with it via tidymodels like so: # define a generalized additive model specification gam_spec <- gen_additive_mod("regression") # fit the specification using a formula with specials fit(gam_spec, mpg ~ cyl + s(disp, k = 5), mtcars) #> parsnip model object #> #> #> Family: gaussian #> Link function: identity #> #> Formula: #> mpg ~ cyl + s(disp, k = 5) #> #> Estimated degrees of freedom: #> 3.39 total = 5.39 #> #> GCV score: 6.380152 While parsnip can handle specials just fine, the package is often used in conjunction with the greater tidymodels package ecosystem, which defines its own pre-processing infrastructure and functionality via packages like hardhat and recipes. The specials defined in many modeling packages introduce conflicts with that infrastructure. To support specials while also maintaining consistent syntax elsewhere in the ecosystem, **tidymodels delineates between two types of formulas: preprocessing formulas and model formulas**. Preprocessing formulas determine the input variables, while model formulas determine the model structure. This is a tricky abstraction, and one that users have tripped up on in the past. Users could generate all sorts of different errors by 1) mistakenly passing model formulas where preprocessing formulas were expected, or 2) forgetting to pass a model formula where it’s needed. For an example of 1), we could pass recipes the same formula we passed to parsnip: recipe(mpg ~ cyl + s(disp, k = 5), mtcars) #> Error in `inline_check()`: #> ! No in-line functions should be used here; use steps to #> define baking actions. But we _just_ used a special with another tidymodels function! Rude! Or, to demonstrate 2), we pass the preprocessing formula as we ought to but forget to provide the model formula: gam_wflow <- workflow() %>% add_formula(mpg ~ .) %>% add_model(gam_spec) gam_wflow %>% fit(mtcars) #> Error in `fit_xy()`: #> ! `fit()` must be used with GAM models (due to its use of formulas). Uh, but I _did_ just use `fit()`! Since the distinction between model formulas and preprocessor formulas comes up in functions across tidymodels, we decide to create a [central page](https://parsnip.tidymodels.org/dev/reference/model_formula.html) that documents the concept itself, hopefully making the syntax associated with it come more easily to users. Then, we link to it _all over the place_. For example, those errors now look like: recipe(mpg ~ cyl + s(disp, k = 5), mtcars) #> Error in `inline_check()`: #> ✖ No in-line functions should be used here. #> ℹ The following function was found: `s`. #> ℹ Use steps to do transformations instead. #> ℹ If your modeling engine uses special terms in formulas, pass that #> formula to workflows as a model formula #> (`?parsnip::model_formula()`). Or: gam_wflow %>% fit(mtcars) #> Error: #> ! When working with generalized additive models, please supply #> the model specification to `workflows::add_model()` along with a #> `formula` argument. #> ℹ See `?parsnip::model_formula()` to learn more. While I’ve only outlined three, there are all sorts of improvements to error messages on their way to the tidymodels packages in upcoming releases. If you happen to stumble across them, we hope they quickly set you back on the right path. 🗺 * * * 1. Issue triage consists of categorizing, prioritizing, and consolidating issues in a repository’s issue tracker. [↩︎](#fnref:1) 2. See the [tidyclust](https://tidyclust.tidymodels.org) package for unsupervised learning with tidymodels! [↩︎](#fnref:2) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dbplyr 2.4.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dbplyr 2.4.0 ============ ![](/blog/2023/10/dbplyr-2-4-0/thumbnail-wd.jpg) Photo by [Parker Hilton](https://unsplash.com/photos/AJqaubLEaN4)   2023/10/26   [dbplyr](/tags/dbplyr/) , [dplyr](/tags/dplyr/)   Hadley Wickham We’re chuffed to announce the release of [dbplyr](http://dbplyr.tidyverse.org/) 2.4.0. dbplyr is a database backend for dplyr that allows you to use a remote database as if it was a collection of local data frames: you write ordinary dplyr code and dbplyr translates it to SQL for you. You can install it from CRAN with: install.packages("dbplyr") This blog post will highlight some of the most important new features: eliminating subqueries when using multiple unions in a row, getting more control on the generated SQL, and a handful of new translations. As usual, release comes with a large number of improvements to translations for individual backends; see the full list in the [release notes](https://github.com/tidyverse/dbplyr/releases/tag/v2.4.0) library(dbplyr) library(dplyr, warn.conflicts = FALSE) SQL optimisation[](#sql-optimisation) -------------------------------------- dbplyr now produces fewer subqueries when combining tables with [`union()`](https://generics.r-lib.org/reference/setops.html) and [`union_all()`](https://dplyr.tidyverse.org/reference/setops.html) resulting in shorter, more readable, and, in some cases, faster SQL. lf1 <- lazy_frame(x = 1, y = "a", .name = "lf1") lf2 <- lazy_frame(x = 1, y = "b", .name = "lf2") lf3 <- lazy_frame(x = 1, z = "c", .name = "lf3") lf1 |> union(lf2) |> union(lf3) #> #> SELECT `lf1`.*, NULL AS `z` #> FROM `lf1` #> #> UNION #> #> SELECT `lf2`.*, NULL AS `z` #> FROM `lf2` #> #> UNION #> #> SELECT `x`, NULL AS `y`, `z` #> FROM `lf3` (As usual in these blog posts, I’m using [`lazy_frame()`](https://dbplyr.tidyverse.org/reference/tbl_lazy.html) to focus on the SQL generation, without having to set up a dummy database.) Similarly, a `semi/anti_join()` on a filtered table now avoids a subquery: lf1 |> semi_join(lf3 |> filter(z == "c"), join_by(x)) #> #> SELECT `lf1`.* #> FROM `lf1` #> WHERE EXISTS ( #> SELECT 1 FROM `lf3` #> WHERE (`lf1`.`x` = `lf3`.`x`) AND (`lf3`.`z` = 'c') #> ) SQL generation[](#sql-generation) ---------------------------------- The new argument `sql_options` for [`show_query()`](https://dplyr.tidyverse.org/reference/explain.html) and [`remote_query()`](https://dbplyr.tidyverse.org/reference/remote_name.html) gives you more control on the generated SQL. * By default dbplyr uses `*` to select all columns of a table, but with `use_star = FALSE` all columns are selected explicitly: lf3 <- lazy_frame(x = 1, y = 2, z = 3, .name = "lf3") lf3 |> mutate(a = 4) #> #> SELECT `lf3`.*, 4.0 AS `a` #> FROM `lf3` lf3 |> mutate(a = 4) |> show_query(sql_options = sql_options(use_star = FALSE)) #> #> SELECT `x`, `y`, `z`, 4.0 AS `a` #> FROM `lf3` * If you prefer common table expressions (CTE) over subqueries use `cte = TRUE`: nested_query <- lf3 |> mutate(z = z + 1) |> left_join(lf2, by = join_by(x, y)) nested_query #> #> SELECT `LHS`.* #> FROM ( #> SELECT `x`, `y`, `z` + 1.0 AS `z` #> FROM `lf3` #> ) AS `LHS` #> LEFT JOIN `lf2` #> ON (`LHS`.`x` = `lf2`.`x` AND `LHS`.`y` = `lf2`.`y`) nested_query |> show_query(sql_options = sql_options(cte = TRUE)) #> #> WITH `q01` AS ( #> SELECT `x`, `y`, `z` + 1.0 AS `z` #> FROM `lf3` #> ) #> SELECT `LHS`.* #> FROM `q01` AS `LHS` #> LEFT JOIN `lf2` #> ON (`LHS`.`x` = `lf2`.`x` AND `LHS`.`y` = `lf2`.`y`) * And if you want that all columns in a join are qualified with the table name and not only the ambiguous ones use `qualify_all_columns = TRUE`: qualify_columns <- lf2 |> left_join(lf3, by = join_by(x, y)) qualify_columns #> #> SELECT `lf2`.*, `z` #> FROM `lf2` #> LEFT JOIN `lf3` #> ON (`lf2`.`x` = `lf3`.`x` AND `lf2`.`y` = `lf3`.`y`) qualify_columns |> show_query(sql_options = sql_options(qualify_all_columns = TRUE)) #> #> SELECT `lf2`.*, `lf3`.`z` AS `z` #> FROM `lf2` #> LEFT JOIN `lf3` #> ON (`lf2`.`x` = `lf3`.`x` AND `lf2`.`y` = `lf3`.`y`) New translations[](#new-translations) -------------------------------------- `str_detect()`, `str_starts()` and `str_ends()` with fixed patterns are translated to `INSTR()`: lf1 |> filter( stringr::str_detect(x, stringr::fixed("abc")), stringr::str_starts(x, stringr::fixed("a")) ) #> #> SELECT `lf1`.* #> FROM `lf1` #> WHERE (INSTR(`x`, 'abc') > 0) AND (INSTR(`x`, 'a') = 1) And [`nzchar()`](https://rdrr.io/r/base/nchar.html) and [`runif()`](https://rdrr.io/r/stats/Uniform.html) are now translated to their SQL equivalents: lf1 |> filter(nzchar(x)) |> mutate(z = runif()) #> #> SELECT `lf1`.*, RANDOM() AS `z` #> FROM `lf1` #> WHERE (((`x` IS NULL) OR `x` != '')) Acknowledgements[](#acknowledgements) -------------------------------------- The vast majority of this release (particularly the SQL optimisations) are from [Maximilian Girlich](https://github.com/mgirlich) ; thanks so much for continued work on this package! And a big thanks go to the 84 other folks who helped out by filing issues and contributing code: [@abalter](https://github.com/abalter) , [@ablack3](https://github.com/ablack3) , [@andreassoteriadesmoj](https://github.com/andreassoteriadesmoj) , [@apalacio9502](https://github.com/apalacio9502) , [@avsdev-cw](https://github.com/avsdev-cw) , [@bairdj](https://github.com/bairdj) , [@bastistician](https://github.com/bastistician) , [@brownj31](https://github.com/brownj31) , [@But2ene](https://github.com/But2ene) , [@carlganz](https://github.com/carlganz) , [@catalamarti](https://github.com/catalamarti) , [@CEH-SLU](https://github.com/CEH-SLU) , [@chriscardillo](https://github.com/chriscardillo) , [@DavisVaughan](https://github.com/DavisVaughan) , [@DaZaM82](https://github.com/DaZaM82) , [@donour](https://github.com/donour) , [@edgararuiz](https://github.com/edgararuiz) , [@eduardszoecs](https://github.com/eduardszoecs) , [@eipi10](https://github.com/eipi10) , [@ejneer](https://github.com/ejneer) , [@erikvona](https://github.com/erikvona) , [@fh-afrachioni](https://github.com/fh-afrachioni) , [@fh-mthomson](https://github.com/fh-mthomson) , [@gui-salome](https://github.com/gui-salome) , [@hadley](https://github.com/hadley) , [@halpo](https://github.com/halpo) , [@homer3018](https://github.com/homer3018) , [@iangow](https://github.com/iangow) , [@jdlom](https://github.com/jdlom) , [@jennal-datacenter](https://github.com/jennal-datacenter) , [@JeremyPasco](https://github.com/JeremyPasco) , [@jiemakel](https://github.com/jiemakel) , [@jingydz](https://github.com/jingydz) , [@johnbaums](https://github.com/johnbaums) , [@joshseiv](https://github.com/joshseiv) , [@jrandall](https://github.com/jrandall) , [@khkk378](https://github.com/khkk378) , [@kmishra9](https://github.com/kmishra9) , [@kongdd](https://github.com/kongdd) , [@krlmlr](https://github.com/krlmlr) , [@krprasangdas](https://github.com/krprasangdas) , [@KRRLP-PL](https://github.com/KRRLP-PL) , [@lentinj](https://github.com/lentinj) , [@lgaborini](https://github.com/lgaborini) , [@lhabegger](https://github.com/lhabegger) , [@lorenzolightsgdwarf](https://github.com/lorenzolightsgdwarf) , [@lschneiderbauer](https://github.com/lschneiderbauer) , [@marianschmidt](https://github.com/marianschmidt) , [@matthewjnield](https://github.com/matthewjnield) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@misea](https://github.com/misea) , [@mjbroerman](https://github.com/mjbroerman) , [@moodymudskipper](https://github.com/moodymudskipper) , [@multimeric](https://github.com/multimeric) , [@nannerhammix](https://github.com/nannerhammix) , [@nikolasharing](https://github.com/nikolasharing) , [@nviets](https://github.com/nviets) , [@nviraj](https://github.com/nviraj) , [@oobd](https://github.com/oobd) , [@pboesu](https://github.com/pboesu) , [@pepijn-devries](https://github.com/pepijn-devries) , [@rbcavanaugh](https://github.com/rbcavanaugh) , [@rcepka](https://github.com/rcepka) , [@robertkck](https://github.com/robertkck) , [@samssann](https://github.com/samssann) , [@SayfSaid](https://github.com/SayfSaid) , [@scottporter](https://github.com/scottporter) , [@shearerpmm](https://github.com/shearerpmm) , [@srikanthtist](https://github.com/srikanthtist) , [@stemangiola](https://github.com/stemangiola) , [@stephenashton-dhsc](https://github.com/stephenashton-dhsc) , [@stevepowell99](https://github.com/stevepowell99) , [@TBlackmore](https://github.com/TBlackmore) , [@thomashulst](https://github.com/thomashulst) , [@thothal](https://github.com/thothal) , [@tilo-aok](https://github.com/tilo-aok) , [@tisseuil](https://github.com/tisseuil) , [@tonyk7440](https://github.com/tonyk7440) , [@TSchiefer](https://github.com/TSchiefer) , [@Tsemharb](https://github.com/Tsemharb) , [@tuge98](https://github.com/tuge98) , [@vadim-cherepanov](https://github.com/vadim-cherepanov) , and [@wdenton](https://github.com/wdenton) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # testthat [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Testthat [![](/blog/2023/10/testthat-3-2-0/thumbnail-sq.jpg)](/blog/2023/10/testthat-3-2-0/) [testthat 3.2.0](/blog/2023/10/testthat-3-2-0/) [package](/categories/package) Hadley Wickham Catch up on the last two years of testthat development which includes improved documentation, new expectations, a new style for error snapshots, support for mocking, a new way to detect if a test has changed global state, and a handful of UI improvements. [Read more ...](/blog/2023/10/testthat-3-2-0/) 2023/10/08 [![](/blog/2022/02/upkeep-testthat-3/thumbnail-sq.jpg)](/blog/2022/02/upkeep-testthat-3/) [Upgrading to testthat edition 3](/blog/2022/02/upkeep-testthat-3/) [learn](/categories/learn) Hannah Frick A workflow for upgrading to testthat edition 3: activation, deprecations, changes to warnings, messages, and comparisons. [Read more ...](/blog/2022/02/upkeep-testthat-3/) 2022/02/22 [![](/blog/2021/10/testthat-3-1/thumbnail-sq.jpg)](/blog/2021/10/testthat-3-1/) [testthat 3.1.0](/blog/2021/10/testthat-3-1/) [package](/categories/package) Hadley Wickham testthat 3.1.0 brings improvements to snapshot testing and one breaking change. [Read more ...](/blog/2021/10/testthat-3-1/) 2021/10/04 [![](/blog/2021/08/waldo-0-3-0/thumbnail-sq.jpg)](/blog/2021/08/waldo-0-3-0/) [waldo 0.3.0](/blog/2021/08/waldo-0-3-0/) [package](/categories/package) [programming](/categories/programming) Hadley Wickham waldo 0.3.0 improves the display of data frame differences, and gives the objects being compared the ability to control the detail of their comparisons. [Read more ...](/blog/2021/08/waldo-0-3-0/) 2021/08/24 [![](/blog/2021/06/vdiffr-1-0-0/thumbnail-sq.jpg)](/blog/2021/06/vdiffr-1-0-0/) [vdiffr 1.0.0](/blog/2021/06/vdiffr-1-0-0/) [package](/categories/package) Lionel Henry This major release of vdiffr includes an updated SVG engine and integrates with the snapshot management mechanism of testthat 3. [Read more ...](/blog/2021/06/vdiffr-1-0-0/) 2021/06/21 [![](/blog/2020/10/testthat-3-0-0/thumbnail-sq.jpg)](/blog/2020/10/testthat-3-0-0/) [testthat 3.0.0](/blog/2020/10/testthat-3-0-0/) [package](/categories/package) Hadley Wickham testhat 3.0.0 brings a raft of major improvements including snapshot testing and parallel testing. It also includes a new “edition” that allows you opt-in to a set of substantial improvements that are not backward compatible. [Read more ...](/blog/2020/10/testthat-3-0-0/) 2020/10/31 [![](/blog/2020/10/waldo/thumbnail-sq.jpg)](/blog/2020/10/waldo/) [waldo](/blog/2020/10/waldo/) [package](/categories/package) Hadley Wickham waldo is a new package that makes it easier to see the differences between a pair of complex R objects. [Read more ...](/blog/2020/10/waldo/) 2020/10/15 [![](/blog/2019/11/testthat-2-3-0/thumb-sq.jpg)](/blog/2019/11/testthat-2-3-0/) [testthat 2.3.0](/blog/2019/11/testthat-2-3-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2019/11/testthat-2-3-0/) 2019/11/06 [![](/blog/2019/04/testthat-2-1-0/testthat-2-1-0-sq.jpg)](/blog/2019/04/testthat-2-1-0/) [testthat 2.1.0](/blog/2019/04/testthat-2-1-0/) [package](/categories/package) Hadley Wickham Highlights include that `context()` is now optional, and new `expect_invisible()` and `expect_mapequal()` expectations. [Read more ...](/blog/2019/04/testthat-2-1-0/) 2019/04/23 [![](/blog/2017/12/testthat-2-0-0/testthat-2-0-0-sq.jpg)](/blog/2017/12/testthat-2-0-0/) [testthat 2.0.0](/blog/2017/12/testthat-2-0-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2017/12/testthat-2-0-0/) 2017/12/19 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Dplyr [![](/blog/2023/10/dbplyr-2-4-0/thumbnail-sq.jpg)](/blog/2023/10/dbplyr-2-4-0/) [dbplyr 2.4.0](/blog/2023/10/dbplyr-2-4-0/) [package](/categories/package) Hadley Wickham dbplyr 2.4.0 brings improvements to SQL generation, better control over the generated SQL, some new translations, and a bunch of backend specific improvements. [Read more ...](/blog/2023/10/dbplyr-2-4-0/) 2023/10/26 [![](/blog/2023/03/dplyr-1-1-1/thumbnail-sq.jpg)](/blog/2023/03/dplyr-1-1-1/) [dplyr 1.1.1](/blog/2023/03/dplyr-1-1-1/) [package](/categories/package) Davis Vaughan dplyr 1.1.1 is on CRAN! This patch release includes a number of performance regression fixes along with refinements to the multiple match join warnings that result in warnings being thrown much less often. [Read more ...](/blog/2023/03/dplyr-1-1-1/) 2023/03/22 [![](/blog/2023/02/dtplyr-1-3-0/thumbnail-sq.jpg)](/blog/2023/02/dtplyr-1-3-0/) [dtplyr 1.3.0](/blog/2023/02/dtplyr-1-3-0/) [package](/categories/package) Hadley Wickham dtplyr brings initial support for dplyr 1.1.0 features, new translations, and a breaking change. [Read more ...](/blog/2023/02/dtplyr-1-3-0/) 2023/02/24 [![](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) [dplyr 1.1.0: `pick()`, `reframe()`, and `arrange()`](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) [package](/categories/package) Davis Vaughan This final post contains a grab-bag of new features, including: `pick()` for column selection inside of data-masking functions, `reframe()` as the new home for `summarise()`'s multi-row behavior, and major performance improvements to `arrange()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) 2023/02/07 [![](/blog/2023/02/dplyr-1-1-0-vctrs/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-vctrs/) [dplyr 1.1.0: The power of vctrs](/blog/2023/02/dplyr-1-1-0-vctrs/) [package](/categories/package) Davis Vaughan All of the dplyr vector functions, like `between()` and `case_when()`, are now powered by vctrs. We’ve also added two powerful new helpers: `case_match()` and `consecutive_id()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-vctrs/) 2023/02/02 [![](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) [dplyr 1.1.0: Per-operation grouping](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) [package](/categories/package) Davis Vaughan dplyr now supports an experimental per-operation grouping syntax. This serves as an alternative to `group_by()` and always returns an ungrouped data frame, meaning that you never need to remember to `ungroup()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) 2023/02/01 [![](/blog/2023/01/dplyr-1-1-0-joins/thumbnail-sq.jpg)](/blog/2023/01/dplyr-1-1-0-joins/) [dplyr 1.1.0: Joins](/blog/2023/01/dplyr-1-1-0-joins/) [package](/categories/package) Davis Vaughan In dplyr 1.1.0, joins have been greatly reworked, including a new way to specify join columns, support for inequality, rolling, and overlap joins, and two new quality control arguments. [Read more ...](/blog/2023/01/dplyr-1-1-0-joins/) 2023/01/31 [![](/blog/2023/01/dbplyr-2-3-0/thumbnail-sq.jpg)](/blog/2023/01/dbplyr-2-3-0/) [dbplyr 2.3.0](/blog/2023/01/dbplyr-2-3-0/) [package](/categories/package) Hadley Wickham dbplyr 2.3.0 brings improvements to SQL generation, improved error messages, a handful of new translations, and a bunch of backend specific improvements. [Read more ...](/blog/2023/01/dbplyr-2-3-0/) 2023/01/16 [![](/blog/2022/11/dplyr-1-1-0-is-coming-soon/thumbnail-sq.jpg)](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) [dplyr 1.1.0 is coming soon](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) [package](/categories/package) Davis Vaughan dplyr 1.1.0 is coming soon! This post introduces some of the exciting new features coming in 1.1.0, and includes a call-for-feedback as we finalize the release. [Read more ...](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) 2022/11/28 [![](/blog/2022/06/dbplyr-2-2-0/thumbnail-sq.jpg)](/blog/2022/06/dbplyr-2-2-0/) [dbplyr 2.2.0](/blog/2022/06/dbplyr-2-2-0/) [package](/categories/package) Hadley Wickham This release brings improvements to SQL translation, a new way of getting local data into the database, and support for dplyr’s family of row modification functions. [Read more ...](/blog/2022/06/dbplyr-2-2-0/) 2022/06/06 * [««](/tags/dplyr/) * « * [1](/tags/dplyr/) * [2](/tags/dplyr/page/2/) * [3](/tags/dplyr/page/3/) * [4](/tags/dplyr/page/4/) * [»](/tags/dplyr/page/2/) * [»»](/tags/dplyr/page/4/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q4 2023 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q4 2023 tidymodels digest ========================= ![](/blog/2024/01/tidymodels-2023-q4/thumbnail-wd.jpg) Photo by [Simon Berger](https://www.pexels.com/photo/landscape-photography-of-snow-pathway-between-trees-during-winter-688660/)   2024/01/09   [tidymodels](/tags/tidymodels/) , [recipes](/tags/recipes/)   Emil Hvitfeldt The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like this post from the past couple of months: * [Three ways errors are about to get better in tidymodels](https://www.tidyverse.org/blog/2023/11/tidymodels-errors-q4/) Since [our last roundup post](https://www.tidyverse.org/blog/2022/12/tidymodels-2022-q4/) , there have been CRAN releases of 7 tidymodels packages. Here are links to their NEWS files: * embed [(1.1.3)](https://embed.tidymodels.org/news/index.html) * modeldb [(0.3.0)](https://modeldb.tidymodels.org/news/index.html) * recipes [(1.0.9)](https://recipes.tidymodels.org/news/index.html) * spatialsample [(0.5.1)](https://spatialsample.tidymodels.org/news/index.html) * stacks [(1.0.3)](https://stacks.tidymodels.org/news/index.html) * textrecipes [(1.0.6)](https://textrecipes.tidymodels.org/news/index.html) * tidyposterior [(1.0.1)](https://tidyposterior.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: updated warnings when normalizing, and better error messages in recipes. library(tidymodels) data("ames", package = "modeldata") Updated warnings when normalizing[](#updated-warnings-when-normalizing) ------------------------------------------------------------------------ The latest release of recipes features an overhaul of the warnings and error messages to use the [cli](https://cli.r-lib.org/) package. With this, we are starting the project of providing more information signaling when things don’t go well. The first type of issue we now signal for is when you try to normalize data that contains elements such as `NA` or `Inf`. These can sneak in for several reasons, and before this release, it happened silently. Below we are creating a recipe using the `ames` data set, and before we normalize, we are taking the logarithms of all variables that pertain to square footage. rec <- recipe(Sale_Price ~ ., data = ames) |> step_log(contains("SF")) |> step_normalize(all_numeric_predictors()) |> prep() #> Warning: Columns `BsmtFin_SF_1`, `BsmtFin_SF_2`, `Bsmt_Unf_SF`, `Total_Bsmt_SF`, #> `Second_Flr_SF`, `Wood_Deck_SF`, and `Open_Porch_SF` returned NaN, because #> variance cannot be calculated and scaling cannot be used. Consider avoiding #> `Inf` or `-Inf` values and/or setting `na_rm = TRUE` before normalizing. We now get a warning that something happened, telling us that it encountered `Inf` or `-Inf`. Knowing that, we can go back and investigate what went wrong. If we exclude `step_normalize()` and `bake()` the recipe, we see that a number of `-Inf` values appear. recipe(Sale_Price ~ ., data = ames) |> step_log(contains("SF")) |> prep() |> bake(new_data = NULL, contains("SF")) |> glimpse() #> Rows: 2,930 #> Columns: 8 #> $ BsmtFin_SF_1 0.6931472, 1.7917595, 0.0000000, 0.0000000, 1.0986123, 1… #> $ BsmtFin_SF_2 -Inf, 4.969813, -Inf, -Inf, -Inf, -Inf, -Inf, -Inf, -Inf… #> $ Bsmt_Unf_SF 6.089045, 5.598422, 6.006353, 6.951772, 4.919981, 5.7807… #> $ Total_Bsmt_SF 6.984716, 6.782192, 7.192182, 7.654443, 6.833032, 6.8308… #> $ First_Flr_SF 7.412160, 6.797940, 7.192182, 7.654443, 6.833032, 6.8308… #> $ Second_Flr_SF -Inf, -Inf, -Inf, -Inf, 6.552508, 6.519147, -Inf, -Inf, … #> $ Wood_Deck_SF 5.347108, 4.941642, 5.973810, -Inf, 5.356586, 5.886104, … #> $ Open_Porch_SF 4.127134, -Inf, 3.583519, -Inf, 3.526361, 3.583519, -Inf… Looking at the bare data set, we notice that the `-Inf` all appear where there are `0`, which makes sense since `log(0)` is undefined. ames |> select(contains("SF")) |> glimpse() #> Rows: 2,930 #> Columns: 8 #> $ BsmtFin_SF_1 2, 6, 1, 1, 3, 3, 3, 1, 3, 7, 7, 1, 7, 3, 3, 1, 3, 3, 4,… #> $ BsmtFin_SF_2 0, 144, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1120, 0, 0, … #> $ Bsmt_Unf_SF 441, 270, 406, 1045, 137, 324, 722, 1017, 415, 994, 763,… #> $ Total_Bsmt_SF 1080, 882, 1329, 2110, 928, 926, 1338, 1280, 1595, 994, … #> $ First_Flr_SF 1656, 896, 1329, 2110, 928, 926, 1338, 1280, 1616, 1028,… #> $ Second_Flr_SF 0, 0, 0, 0, 701, 678, 0, 0, 0, 776, 892, 0, 676, 0, 0, 1… #> $ Wood_Deck_SF 210, 140, 393, 0, 212, 360, 0, 0, 237, 140, 157, 483, 0,… #> $ Open_Porch_SF 62, 0, 36, 0, 34, 36, 0, 82, 152, 60, 84, 21, 75, 0, 54,… Knowing that it was `0` that caused the problem, we can set an `offset` to avoid taking `log(0)`. rec <- recipe(Sale_Price ~ ., data = ames) |> step_log(contains("SF"), offset = 0.5) |> step_normalize(all_numeric_predictors()) |> prep() These warnings appear in `step_scale()`, `step_normalize()`, `step_center()` or `step_range()`. Better error messages in recipes[](#better-error-messages-in-recipes) ---------------------------------------------------------------------- Another problem that happens a lot when using recipes, is accidentally selecting variables that have the wrong types. Previously this caused the following error: recipe(Sale_Price ~ ., data = ames) |> step_dummy(starts_with("Lot_")) |> prep() #> Error in `step_dummy()`: #> Caused by error in `prep()`: #> ! All columns selected for the step should be string, factor, or ordered. In the newest release, it will detail the offending variables and what was wrong with them. recipe(Sale_Price ~ ., data = ames) |> step_dummy(starts_with("Lot_")) |> prep() |> bake() #> Error in `step_dummy()`: #> Caused by error in `prep()`: #> ✖ All columns selected for the step should be factor or ordered. #> • 1 double variable found: `Lot_Frontage` #> • 1 integer variable found: `Lot_Area` Coming Attractions[](#coming-attractions) ------------------------------------------ In the next month or so we are planning a cascade of CRAN releases. There is a lot of new functionality coming your way, especially in the tune package. A number of our packages will (finally) be able to cohesively fit, evaluate, tune, and predict models for event times (a.k.a., [survival analysis](https://en.wikipedia.org/wiki/Survival_analysis) ). If you don’t do this type of work, you might not notice the new capabilities. However, if you do, tidymodels will be able to do a lot more for you. We’ve also implemented a number of features related to model fairness. These tools allow tidymodels users to identify when machine learning models behave unfairly towards certain groups of people, and will also be included in the upcoming releases of tidymodels packages in Q1. We’ll highlight a lot of these new capabilities in blog posts here as well as tutorials on [`tidymodels.org`](https://www.tidymodels.org/) . So, there’s a lot more coming! We are very excited to have these features officially available and to see what people can do with them. Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to thank those in the community that contributed to tidymodels in the last quarter: * embed: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) . * modeldb: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hadley](https://github.com/hadley) , and [@topepo](https://github.com/topepo) . * recipes: [@atusy](https://github.com/atusy) , [@bcadenato](https://github.com/bcadenato) , [@collinberke](https://github.com/collinberke) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gfronk](https://github.com/gfronk) , [@jkennel](https://github.com/jkennel) , [@joeycouse](https://github.com/joeycouse) , [@jxu](https://github.com/jxu) , [@mastoffel](https://github.com/mastoffel) , [@matthewgson](https://github.com/matthewgson) , [@millermc38](https://github.com/millermc38) , [@ray-p144](https://github.com/ray-p144) , [@sebsfox](https://github.com/sebsfox) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * spatialsample: [@mikemahoney218](https://github.com/mikemahoney218) . * stacks: [@juliasilge](https://github.com/juliasilge) , and [@simonpcouch](https://github.com/simonpcouch) . * textrecipes: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@jd4ds](https://github.com/jd4ds) , and [@masurp](https://github.com/masurp) . * tidyposterior: [@topepo](https://github.com/topepo) . We’re grateful for all of the tidymodels community, from observers to users to contributors. Happy modeling! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # scales 1.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) scales 1.3.0 ============ ![](/blog/2023/11/scales-1-3-0/thumbnail-wd.jpg) Photo by [Elena Mozhvilo](https://unsplash.com/photos/gold-and-silver-round-frame-magnifying-glass-j06gLuKK0GM)   2023/11/27   [scales](/tags/scales/)   Thomas Lin Pedersen We’re delighted to announce the release of [scales](https://scales.r-lib.org) 1.3.0. scales is a packages that extracts much of the scaling logic that is used in ggplot2 to a general framework, along with utility functions for e.g. formatting labels or creating color palettes. You can install it from CRAN with: install.packages("scales") This blog post will give a quick overview of the 1.3.0 release, which is mainly an upkeep release but does contain a few interesting tidbits. You can see a full list of changes in the [release notes](https://scales.r-lib.org/news/index.html) library(scales) set.seed(1) Proper support for difftime objects[](#proper-support-for-difftime-objects) ---------------------------------------------------------------------------- While scales had rudimentary support for objects from the hms package, I did not support the more common base R difftime objects. This is now rectified with the introduction of [`label_timespan()`](https://scales.r-lib.org/reference/label_date.html) , [`breaks_timespan()`](https://scales.r-lib.org/reference/breaks_timespan.html) , and [`transform_timespan()`](https://scales.r-lib.org/reference/transform_timespan.html) . While the labels and breaks function can be used on their own, all the behavior is encapsulated in the timespan transform object which is kin to [`transform_hms()`](https://scales.r-lib.org/reference/transform_timespan.html) . library(ggplot2) events <- data.frame( time = as.difftime(runif(30, max = 200), units = "secs"), magnitude = rnorm(30) + 2 ) ggplot(events) + geom_point( aes(time, y = 0, size = magnitude), position = position_jitter(width = 0) ) + scale_x_continuous(trans = transform_timespan()) ![](figs/unnamed-chunk-2-1.png) As we can see the timespan transform automatically picks the unit of the difftime object. Further it identifies that for this range, adding breaks for minutes makes most sense. If we had recorded time as hours rather than seconds, we can see how that affects the labelling: events$time <- as.difftime(runif(30, max = 200), units = "hours") ggplot(events) + geom_point( aes(time, y = 0, size = magnitude), position = position_jitter(width = 0) ) + scale_x_continuous(trans = transform_timespan()) ![](figs/unnamed-chunk-3-1.png) API brush-up[](#api-brush-up) ------------------------------ scales has gone through a number of touch-ups on its API, such as revamping the labels functions to all start with `label_`. This release we continue (and hopefully conclude) the touch-ups by using a common prefix for the transformation utilities (`transform_`) and palettes (`pal_`). We have also rename [`label_dollar()`](https://scales.r-lib.org/reference/dollar_format.html) to [`label_currency()`](https://scales.r-lib.org/reference/label_currency.html) to make it clear that this can be used for any type of currency, not just dollars (US or otherwise). All the old functions have been kept around with no plan of deprecation but we advise you to update your code to use the new names. More transformation power[](#more-transformation-power) -------------------------------------------------------- This release also includes some other updates to the transformations. They have received a fair amount of bug fixes and a new built-in transformation type has joined the group: [`transform_asinh()`](https://scales.r-lib.org/reference/transform_asinh.html) , the inverse hyperbolic sine transformation, can be used much like log transformations, but it also supports negative values. plot(transform_asinh(), xlim = c(-100, 100)) lines(seq(-100, 100), transform_log()$transform(seq(-100, 100)), col = "red") text(50, 3, label = "log-transform", col = "red", adj = 0) ![](figs/unnamed-chunk-4-1.png) Transformation objects can now also (optionally) record the derivatives and inverse derivative which makes it possible to properly correct density estimations of transformed values. Fixes to range training in discrete scales[](#fixes-to-range-training-in-discrete-scales) ------------------------------------------------------------------------------------------ The training of discrete ranges has seen a few changes that hopefully makes it more predictable what happens when you train a range based on factors or character vectors. When training based on factors the ordering of the range will follow the order of the levels in the factor as they are encountered. New values will be appended to the end of the range. For character vectors the range will always stay sorted alphanumerically. Mixing of character and factors during training will lead to undefined ordering. This has always been the advertised behavior but it was not applied consistently up until now. As a result you may see the occational reordering of e.g. legends in ggplot2 after upgrading scales. Acknowledgements[](#acknowledgements) -------------------------------------- [@AndreeWarby](https://github.com/AndreeWarby) , [@ari-nz](https://github.com/ari-nz) , [@BioinformaNicks](https://github.com/BioinformaNicks) , [@bwiernik](https://github.com/bwiernik) , [@ccsarapas](https://github.com/ccsarapas) , [@CMKnott](https://github.com/CMKnott) , [@DanChaltiel](https://github.com/DanChaltiel) , [@davidhodge931](https://github.com/davidhodge931) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dmurdoch](https://github.com/dmurdoch) , [@EricMarcon](https://github.com/EricMarcon) , [@Generalized](https://github.com/Generalized) , [@hadley](https://github.com/hadley) , [@JJHelly](https://github.com/JJHelly) , [@joshuaylevy](https://github.com/joshuaylevy) , [@jzadra](https://github.com/jzadra) , [@kuriwaki](https://github.com/kuriwaki) , [@larmarange](https://github.com/larmarange) , [@laurejo1](https://github.com/laurejo1) , [@lz1nwm](https://github.com/lz1nwm) , [@MikkoVihtakari](https://github.com/MikkoVihtakari) , [@mjskay](https://github.com/mjskay) , [@pearsonca](https://github.com/pearsonca) , [@Saadi4469](https://github.com/Saadi4469) , [@teunbrand](https://github.com/teunbrand) , [@thomasp85](https://github.com/thomasp85) , and [@zeehio](https://github.com/zeehio) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # withr 3.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) withr 3.0.0 =========== ![](/blog/2024/01/withr-3-0-0/thumbnail-wd.jpg) Photo by [Neal E. Johnson](https://unsplash.com/photos/brown-and-black-brush-on-brown-wooden-table-V0cSTljC92k)   2024/01/18   [r-lib](/tags/r-lib/) , [withr](/tags/withr/)   Lionel Henry It’s not without jubilant bearing that we announce the release of the 3.0.0 version of [withr](https://withr.r-lib.org/) , the tidyverse solution for automatic cleanup of resources! In this release, the internals of withr were rewritten to improve the performance and increase the compatibility with base R’s [`on.exit()`](https://rdrr.io/r/base/on.exit.html) mechanism. You can install it from CRAN with: install.packages("withr") In this blog post we’ll go over the changes that made this rewrite possible, but first we’ll review the cleanup strategies made possible by withr. You can see a full list of changes in the [release notes](https://withr.r-lib.org/news/index.html#withr-300) . Cleaning up resources with base R and with withr[](#cleaning-up-resources-with-base-r-and-with-withr) ------------------------------------------------------------------------------------------------------ Traditionally, resource cleanup in R is done with [`base::on.exit()`](https://rdrr.io/r/base/on.exit.html) . Cleaning up in the on-exit hook ensures that the cleanup happens both in the normal case, when the code has finished running without error, and in the error case, when something went wrong and execution is interrupted. [`on.exit()`](https://rdrr.io/r/base/on.exit.html) is meant to be used inside functions but it also works within [`local()`](https://rdrr.io/r/base/eval.html) , which we’ll use here for our examples: local({ on.exit(message("Cleaning time!")) print(1 + 2) }) #> [1] 3 #> Cleaning time! local({ on.exit(message("Cleaning time!")) stop("uh oh") print(1 + 2) }) #> Error: #> ! uh oh #> Cleaning time! [`on.exit()`](https://rdrr.io/r/base/on.exit.html) is guaranteed to run no matter what and this property makes it invaluable for resource cleaning. No more accidental littering! However the process of cleaning up this way can be a bit verbose and feel too manual. Here is how you’d create and clean up a temporary file for instance: local({ my_file <- tempfile() file.create(my_file) on.exit(file.remove(my_file)) writeLines(c("a", "b"), con = my_file) }) Wouldn’t it be great if we could wrap this code up in a function? That’s the goal of withr’s `local_`\-prefixed functions. They combine both the creation or modification of a resource and its (eventual) restoration to the original state into a single function: local({ my_file <- withr::local_tempfile() writeLines(c("a", "b"), con = my_file) }) In this case we have created a resource (a file), but the same principle applies to modifying resources such as global options: local({ # Let's temporarily print with a single decimal place withr::local_options(digits = 1) print(1/3) }) #> [1] 0.3 # The original option value has been restored getOption("digits") #> [1] 7 print(1/3) #> [1] 0.3333333 And you can equivalently use the `with_`\-prefixed variants (from which the package takes its name!), this way you don’t need to wrap in [`local()`](https://rdrr.io/r/base/eval.html) : withr::with_options(list(digits = 1), print(1/3)) #> [1] 0.3 The `with_` functions are useful for creating very small scopes for given resources, inside or outside a function. The withr 3.0.0 rewrite[](#the-withr-300-rewrite) -------------------------------------------------- Traditionally, withr implemented its own exit event system on top of [`on.exit()`](https://rdrr.io/r/base/on.exit.html) . We needed an extra layer because of a couple of missing features: * When multiple resources are managed by a piece of code, the order in which these resources are restored or cleaned up sometimes matter. The most consistent order for cleanup is last-in first-out (LIFO). In other words the oldest resource, on which younger resources might depend, is cleaned up last. But historically R only supported first-in first-out (FIFO) order. * The other missing piece was being able to inspect the contents of the exit hook. The [`sys.on.exit()`](https://rdrr.io/r/base/sys.parent.html) R helper was created for this purpose but was affected by a bug that prevented it from working inside functions. We contributed two changes to R 3.5.0 that filled these missing pieces, fixing the [`sys.on.exit()`](https://rdrr.io/r/base/sys.parent.html) bug and adding an `after` argument to [`on.exit()`](https://rdrr.io/r/base/on.exit.html) to allow last-in first-out ordering. Until now, we haven’t been able to leverage these contributions because of our policy of [supporting the current and previous four versions of R](https://www.tidyverse.org/blog/2019/04/r-version-support) . Now that enough time has passed, it was time for a rewrite! Our version of [`base::on.exit()`](https://rdrr.io/r/base/on.exit.html) is [`withr::defer()`](https://withr.r-lib.org/reference/defer.html) . Along with better default behaviour, [`withr::defer()`](https://withr.r-lib.org/reference/defer.html) allows the clean up of resources non-locally (ironically an essential feature for implementing `local_` functions). Given the changes in R 3.5.0, [`withr::defer()`](https://withr.r-lib.org/reference/defer.html) can now be implemented as a simple wrapper around [`on.exit()`](https://rdrr.io/r/base/on.exit.html) . One benefit of the rewrite is that mixing withr tools and [`on.exit()`](https://rdrr.io/r/base/on.exit.html) in the same function now correctly interleaves cleanup: local({ on.exit(print(1)) withr::defer(print(2)) on.exit(print(3), add = TRUE, after = FALSE) withr::defer(print(4)) print(5) }) #> [1] 5 #> [1] 4 #> [1] 3 #> [1] 2 #> [1] 1 But the main benefit is increased performance. Here is how `defer()` compared to [`on.exit()`](https://rdrr.io/r/base/on.exit.html) in the previous version: base <- function() on.exit(NULL) withr <- function() defer(NULL) # withr 2.5.2 bench::mark(base(), withr(), check = FALSE)[1:8] #> # A tibble: 2 × 8 #> expression min median `itr/sec` mem_alloc `gc/sec` n_itr n_gc #> #> 1 base() 0 82ns 6954952. 0B 696. 9999 1 #> 2 withr() 26.2µs 27.9µs 35172. 88.4KB 52.8 9985 15 withr 3.0.0 has now caught up to [`on.exit()`](https://rdrr.io/r/base/on.exit.html) quite a bit: # withr 3.0.0 bench::mark(base(), withr(), check = FALSE)[1:8] #> # A tibble: 2 × 8 #> expression min median `itr/sec` mem_alloc `gc/sec` n_itr n_gc #> #> 1 base() 0 82ns 7329829. 0B 0 10000 0 #> 2 withr() 2.95µs 3.4µs 280858. 0B 225. 9992 8 Of course [`on.exit()`](https://rdrr.io/r/base/on.exit.html) is still much faster, in part because `defer()` supports more features (more on that below), but mostly because `on.exit` is a primitive function whereas `defer()` is implemented as a normal R function. That said, we hope that we now have made `defer()` (and the `local_` and `with_` functions that use it) sufficiently fast to be used even in performance-critical micro-tools. Improved withr features[](#improved-withr-features) ---------------------------------------------------- Over the successive releases of withr we’ve improved the behaviour of cleanup expressions interactively, in scripts executed with [`source()`](https://rdrr.io/r/base/source.html) , and in knitr. [`on.exit()`](https://rdrr.io/r/base/on.exit.html) is a bit inconsistent when it is used outside of a function: * Interactively, it doesn’t do anything. * In [`source()`](https://rdrr.io/r/base/source.html) and in knitr, it runs immediately instead of a the end of the script [`withr::defer()`](https://withr.r-lib.org/reference/defer.html) and the [`withr::local_`](https://withr.r-lib.org/reference/with_.html) helpers try to be more helpful for these cases. Interactively, it saves the cleanup action in a special global hook and you get information about how to actually perform the cleanup: file <- withr::local_tempfile() #> Setting global deferred event(s). #> i These will be run: #> * Automatically, when the R session ends. #> * On demand, if you call `withr::deferred_run()`. #> i Use `withr::deferred_clear()` to clear them without executing. # Clean up now withr::deferred_run() #> Ran 1/1 deferred expressions In knitr or [`source()`](https://rdrr.io/r/base/source.html) [1](#fn:1) , the cleanup is performed at the end of the document or of the script. If you need chunk-level cleanup, use [`local()`](https://rdrr.io/r/base/eval.html) as we’ve been doing in the examples of this blog post: Cleaning up at the end of the document: ```r document_wide_file <- withr::local_tempfile() ``` Cleaning up at the end of the chunk: ```r local({ local_file <- withr::local_tempfile() }) ``` Starting from withr 3.0.0, you can also run `deferred_run()` inside of a chunk: ```r withr::deferred_run() #> Ran 1/1 deferred expressions ``` Acknowledgements[](#acknowledgements) -------------------------------------- Thanks to the github contributors who helped us with this release! [@ashbythorpe](https://github.com/ashbythorpe) , [@bastistician](https://github.com/bastistician) , [@DavisVaughan](https://github.com/DavisVaughan) , [@fkohrt](https://github.com/fkohrt) , [@gaborcsardi](https://github.com/gaborcsardi) , [@gdurif](https://github.com/gdurif) , [@hadley](https://github.com/hadley) , [@HenrikBengtsson](https://github.com/HenrikBengtsson) , [@honghaoli42](https://github.com/honghaoli42) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@jameslairdsmith](https://github.com/jameslairdsmith) , [@jennybc](https://github.com/jennybc) , [@jonkeane](https://github.com/jonkeane) , [@krlmlr](https://github.com/krlmlr) , [@lionel-](https://github.com/lionel-) , [@maelle](https://github.com/maelle) , [@MichaelChirico](https://github.com/MichaelChirico) , [@MLopez-Ibanez](https://github.com/MLopez-Ibanez) , [@moodymudskipper](https://github.com/moodymudskipper) , [@multimeric](https://github.com/multimeric) , [@orichters](https://github.com/orichters) , [@pfuehrlich-pik](https://github.com/pfuehrlich-pik) , [@solmos](https://github.com/solmos) , [@tillea](https://github.com/tillea) , and [@vanhry](https://github.com/vanhry) . * * * 1. [`source()`](https://rdrr.io/r/base/source.html) is only supported by default when running in the global environment, which is usually the case. For the special case of sourcing in a local environment, you need to set `options(withr.hook_source = TRUE)` first. [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # rsample [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Rsample [![](/blog/2023/10/tidymodels-2023-q3/thumbnail-sq.jpg)](/blog/2023/10/tidymodels-2023-q3/) [Q3 2023 tidymodels digest](/blog/2023/10/tidymodels-2023-q3/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/10/tidymodels-2023-q3/) 2023/10/05 [![](/blog/2023/08/validation-split-as-3-way-split/thumbnail-sq.jpg)](/blog/2023/08/validation-split-as-3-way-split/) [New interface to validation splits](/blog/2023/08/validation-split-as-3-way-split/) [package](/categories/package) Hannah Frick The latest releases of rsample and tune provide a new interface to validation sets as a three-way split. [Read more ...](/blog/2023/08/validation-split-as-3-way-split/) 2023/08/25 [![](/blog/2022/12/tidymodels-2022-q4/thumbnail-sq.jpg)](/blog/2022/12/tidymodels-2022-q4/) [Q4 2022 tidymodels digest](/blog/2022/12/tidymodels-2022-q4/) [roundup](/categories/roundup) Simon Couch The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2022/12/tidymodels-2022-q4/) 2022/12/29 [![](/blog/2022/08/rsample-1-1-0/thumbnail-sq.jpg)](/blog/2022/08/rsample-1-1-0/) [rsample 1.1.0](/blog/2022/08/rsample-1-1-0/) [package](/categories/package) Mike Mahoney rsample 1.1.0 is now on CRAN! This release provides a ton of new functions for grouped resampling, as well as a few long-awaited utility functions. [Read more ...](/blog/2022/08/rsample-1-1-0/) 2022/08/08 [![](/blog/2022/06/spatialsample-0-2-0/thumbnail-sq.jpg)](/blog/2022/06/spatialsample-0-2-0/) [spatialsample 0.2.0](/blog/2022/06/spatialsample-0-2-0/) [package](/categories/package) Mike Mahoney spatialsample 0.2.0 is now on CRAN! This release provides a bunch of new features, including new spatial resampling methods, visualization helpers, and spatial buffering. [Read more ...](/blog/2022/06/spatialsample-0-2-0/) 2022/06/21 [![](/blog/2021/12/tidymodels-2021-q4/thumbnail-sq.jpg)](/blog/2021/12/tidymodels-2021-q4/) [Closing out our year with a Q4 2021 tidymodels update](/blog/2021/12/tidymodels-2021-q4/) [roundup](/categories/roundup) Julia Silge New features like an R Markdown template and Shiny app offer more robust support for modeling analyses, and our survey results outline some priorities for next year. [Read more ...](/blog/2021/12/tidymodels-2021-q4/) 2021/12/16 [![](/blog/2021/03/tidymodels-2021-q1/thumbnail-sq.jpg)](/blog/2021/03/tidymodels-2021-q1/) [Catch up with tidymodels](/blog/2021/03/tidymodels-2021-q1/) [roundup](/categories/roundup) Julia Silge Releases of tidymodels packages in Q1 of 2021 offer new functions for easier model building and resampling, along with a new package for resampling spatial data. [Read more ...](/blog/2021/03/tidymodels-2021-q1/) 2021/03/10 [![](/blog/2019/09/tidymodels-2019-09/tidymodels-2019-09-sq.jpg)](/blog/2019/09/tidymodels-2019-09/) [tidymodels updates](/blog/2019/09/tidymodels-2019-09/) [package](/categories/package) Max Kuhn, Edgar Ruiz, and Davis Vaughan The latest updates to the tidymodels packages [Read more ...](/blog/2019/09/tidymodels-2019-09/) 2019/09/05 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyclust [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyclust [![](/blog/2023/10/tidymodels-2023-q3/thumbnail-sq.jpg)](/blog/2023/10/tidymodels-2023-q3/) [Q3 2023 tidymodels digest](/blog/2023/10/tidymodels-2023-q3/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/10/tidymodels-2023-q3/) 2023/10/05 [![](/blog/2022/12/tidyclust-0-1-0/thumbnail-sq.jpg)](/blog/2022/12/tidyclust-0-1-0/) [tidyclust is on CRAN](/blog/2022/12/tidyclust-0-1-0/) [package](/categories/package) Emil Hvitfeldt Tidyclust is on CRAN. tidyclust provides a common interface for specifying clustering models, in the same style as parsnip. [Read more ...](/blog/2022/12/tidyclust-0-1-0/) 2022/12/06 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q3 2023 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q3 2023 tidymodels digest ========================= ![](/blog/2023/10/tidymodels-2023-q3/thumbnail-wd.jpg) Photo by [Tai's Captures](https://unsplash.com/photos/PGRwUQQhzMQ)   2023/10/05   [tidymodels](/tags/tidymodels/) , [rsample](/tags/rsample/) , [tidyclust](/tags/tidyclust/)   Emil Hvitfeldt The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like this post from the past couple of months: * [New interface to validation splits](https://www.tidyverse.org/blog/2023/08/validation-split-as-3-way-split/) Since [our last roundup post](https://www.tidyverse.org/blog/2022/12/tidymodels-2022-q4/) , there have been CRAN releases of 11 tidymodels packages. Here are links to their NEWS files: * butcher [(0.3.3)](https://butcher.tidymodels.org/news/index.html) * embed [(1.1.2)](https://embed.tidymodels.org/news/index.html) * modeldata [(1.2.0)](https://modeldata.tidymodels.org/news/index.html) * parsnip [(1.1.1)](https://parsnip.tidymodels.org/news/index.html) * recipes [(1.0.8)](https://recipes.tidymodels.org/news/index.html) * rsample [(1.2.0)](https://rsample.tidymodels.org/news/index.html) * textrecipes [(1.0.4)](https://textrecipes.tidymodels.org/news/index.html) * themis [(1.0.2)](https://themis.tidymodels.org/news/index.html) * tidyclust [(0.2.0)](https://tidyclust.tidymodels.org/news/index.html) * tidymodels [(1.1.1)](https://tidymodels.tidymodels.org/news/index.html) * tune [(1.1.2)](https://tune.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: Updated workshop material, new K-means engines and quality of life improvements in rsample. First, loading the collection of packages: library(tidymodels) library(tidyclust) data("ames", package = "modeldata") Workshops[](#workshops) ------------------------ One of the biggest areas of work for our team this quarter was getting ready for this year’s [posit::conf](https://posit.co/conference/) . This year, two 1-day workshops were available: “Introduction to tidymodels” and “Advanced tidymodels”. All the material can be found on our workshop website [workshops.tidymodels.org](https://workshops.tidymodels.org/) , with these workshops being archived as [posit::conf 2023 workshops](https://workshops.tidymodels.org/archive/2023-09-posit-conf/) . Unless otherwise noted (i.e. not an original creation and reused from another source), these educational materials are licensed under Creative Commons Attribution [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) . Tidyclust update[](#tidyclust-update) -------------------------------------- The latest release of tidyclust featured a round of bug fixes, documentation improvements and quality-of-life improvements. This release adds 2 new engines to the [`k_means()`](https://tidyclust.tidymodels.org/reference/k_means.html) model. [klaR](https://tidyclust.tidymodels.org/reference/details_k_means_klaR.html) to run K-Modes models and [clustMixType](https://tidyclust.tidymodels.org/reference/details_k_means_clustMixType.html) to run K-prototypes. K-Modes is the categorical analog to K-means, meaning that it is intended to be used on only categorical data, and K-prototypes is the more general method that works with categorical and numeric data at the same time. If we were to fit a K-means model to a mixed-type data set such as `ames`, it would work, but under the hood, the model would apply a dummy transformation on the categorical predictors. kmeans_spec <- k_means(num_clusters = 3) %>% set_engine("stats") kmeans_fit <- kmeans_spec %>% fit(~ ., data = ames) When extracting the cluster means, we see that the dummy variables were used when calculating the means, which can make it harder to interpret the output. kmeans_fit %>% extract_centroids() %>% select(101:112) %>% glimpse() #> Rows: 3 #> Columns: 12 #> $ Overall_CondGood 0.09009009, 0.17594787, 0.01234568 #> $ Overall_CondVery_Good 0.02702703, 0.06694313, 0.01646091 #> $ Overall_CondExcellent 0.01201201, 0.01303318, 0.02880658 #> $ Overall_CondVery_Excellent 0, 0, 0 #> $ Year_Built 1989.645, 1956.471, 1999.572 #> $ Year_Remod_Add 1996.090, 1974.518, 2003.379 #> $ Roof_StyleGable 0.8238238, 0.8234597, 0.4444444 #> $ Roof_StyleGambrel 0.005005005, 0.010071090, 0.000000000 #> $ Roof_StyleHip 0.1531532, 0.1558057, 0.5555556 #> $ Roof_StyleMansard 0.005005005, 0.003554502, 0.000000000 #> $ Roof_StyleShed 0.003003003, 0.001184834, 0.000000000 #> $ Roof_MatlCompShg 0.9759760, 0.9905213, 0.9876543 Fitting a K-prototype model is done by setting the engine in [`k_means()`](https://tidyclust.tidymodels.org/reference/k_means.html) to `"clustMixType"`. kproto_spec <- k_means(num_clusters = 3) %>% set_engine("clustMixType") kproto_fit <- kproto_spec %>% fit(~ ., data = ames) The clusters can now be extracted on the original data format as categorical predictors are supported. kproto_fit %>% extract_centroids() %>% select(11:20) %>% glimpse() #> Rows: 3 #> Columns: 10 #> $ Lot_Config Inside, Inside, Inside #> $ Land_Slope Gtl, Gtl, Gtl #> $ Neighborhood College_Creek, North_Ames, Northridge_Heights #> $ Condition_1 Norm, Norm, Norm #> $ Condition_2 Norm, Norm, Norm #> $ Bldg_Type OneFam, OneFam, OneFam #> $ House_Style Two_Story, One_Story, One_Story #> $ Overall_Cond Average, Average, Average #> $ Year_Built 1989.977, 1953.793, 1998.765 #> $ Year_Remod_Add 1995.934, 1972.973, 2003.035 Stricter rsample functions[](#stricter-rsample-functions) ---------------------------------------------------------- Before version 1.2.0 of rsample, misspelled and wrongly used arguments would be swallowed silently by the functions. This could be a big source of confusion as it is easy to slip between the cracks. We have made changes to all rsample functions such that whenever possible they alert the user when something is wrong. Before 1.2.0 when you, for example, misspelled `strata` as `stata`, everything would go on like normal, with no indication that `stata` was ignored. initial_split(ames, prop = 0.75, stata = Neighborhood) #> #> <2197/733/2930> The same code will now error and point to the problematic arguments. initial_split(ames, prop = 0.75, stata = Neighborhood) #> Error in `initial_split()`: #> ! `...` must be empty. #> ✖ Problematic argument: #> • stata = Neighborhood Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to thank those in the community that contributed to tidymodels in the last quarter: * butcher: [@hfrick](https://github.com/hfrick) , and [@juliasilge](https://github.com/juliasilge) . * embed: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@wbuchanan](https://github.com/wbuchanan) . * modeldata: [@topepo](https://github.com/topepo) . * parsnip: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gmcmacran](https://github.com/gmcmacran) , [@SHo-JANG](https://github.com/SHo-JANG) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@vidarsumo](https://github.com/vidarsumo) . * recipes: [@abichat](https://github.com/abichat) , [@andreranza](https://github.com/andreranza) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@jkennel](https://github.com/jkennel) , [@millermc38](https://github.com/millermc38) , [@nikosGeography](https://github.com/nikosGeography) , [@pgg1309](https://github.com/pgg1309) , [@rdavis120](https://github.com/rdavis120) , [@Sade154](https://github.com/Sade154) , [@topepo](https://github.com/topepo) , and [@walrossker](https://github.com/walrossker) . * rsample: [@godscloset](https://github.com/godscloset) , [@hfrick](https://github.com/hfrick) , [@MasterLuke84](https://github.com/MasterLuke84) , [@mikemahoney218](https://github.com/mikemahoney218) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * textrecipes: [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@gaohuachuan](https://github.com/gaohuachuan) . * themis: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) . * tidyclust: [@coforfe](https://github.com/coforfe) , [@cphaarmeyer](https://github.com/cphaarmeyer) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@michaelgrund](https://github.com/michaelgrund) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , and [@trevorcampbell](https://github.com/trevorcampbell) . * tidymodels: [@nikosGeography](https://github.com/nikosGeography) , and [@topepo](https://github.com/topepo) . * tune: [@dramanica](https://github.com/dramanica) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@forecastingEDs](https://github.com/forecastingEDs) , [@hfrick](https://github.com/hfrick) , [@kbodwin](https://github.com/kbodwin) , [@KJT-Habitat](https://github.com/KJT-Habitat) , [@MasterLuke84](https://github.com/MasterLuke84) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . We’re grateful for all of the tidymodels community, from observers to users to contributors. Happy modeling! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # httr2 1.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) httr2 1.0.0 =========== ![](/blog/2023/11/httr2-1-0-0/thumbnail-wd.jpg) Photo by [Mike Bowman](https://unsplash.com/photos/xKShyIiTNJk)   2023/11/14   [httr2](/tags/httr2/) , [httr](/tags/httr/)   Hadley Wickham We’re delighted to announce the release of [httr2](https://httr2.r-lib.org) [1](#fn:1) 1.0.0. httr2 is the second generation of httr: it helps you generate HTTP requests and process the responses, designed with an eye towards modern web APIs and potentially putting your code in a package. You can install it from CRAN with: install.packages("httr2") httr2 has been under development for the last two years, but this is the first time we’ve blogged about it because we’ve been waiting until the user interface felt stable. It now does, and we’re ready to encourage you to use httr2 whenever you need to talk to a web server. Most importantly httr2 is now a “real” package because it has a wonderful new logo, thanks to a collaborative effort involving Julie Jung, Greg Swineheart, and DALL•E 3. ![The new httr2 logo is a dark blue hexagon with httr2 written in bright white at the top of logo. Underneath the text is a vibrant magenta baseball player hitting a ball emblazoned with the letters "www".](httr2.png) httr2 is the successor to httr. The biggest difference is that it has an explicit request object which you can build up over multiple function calls. This makes the interface fit more naturally with the pipe, and generally makes life easier because you can iteratively build up a complex request. httr2 also builds on the 10 years of package development experience we’ve accrued since creating httr, so it should all around be more enjoyable to use. If you’re a current httr user, there’s no need to switch, as we’ll continue to maintain the package for many years to come, but if you start on a new project, I’d recommend that you give httr2 a shot. If you’ve been following httr2 development for a while, you might want to jump to the [release notes](https://github.com/r-lib/httr2/releases/tag/v1.0.0) to see what’s new (a lot!). The most important change in this release is that [Maximilian Girlich](https://github.com/mgirlich) is now a httr2 author, in recognition of his many contributions to the package. This release also features improved tools for performing multiple requests (more on that below) and a bunch of bug fixes and minor improvements for OAuth. For the rest of this blog post, I’ll assume that you’re familiar with the basics of HTTP. If you’re not, you might want to start with `vignette("httr2")` which introduces you to HTTP using httr2. library(httr2) Making a request[](#making-a-request) -------------------------------------- httr2 is designed around the two big pieces of HTTP: requests and responses. First you’ll create a request, with a URL: req <- request(example_url()) req #> #> GET http://127.0.0.1:51981/ #> Body: empty Instead of using an external website, here we’re using a test server that’s built in to httr2. This ensures that this blog post, and many httr2 examples, work independently from the rest of the internet. You can see the HTTP request that httr2 will send, without actually sending it[2](#fn:2) , by doing a dry run: req |> req_dry_run() #> GET / HTTP/1.1 #> Host: 127.0.0.1:51981 #> User-Agent: httr2/0.2.3.9000 r-curl/5.1.0 libcurl/8.1.2 #> Accept: */* #> Accept-Encoding: deflate, gzip As you can see, this request object will perform a simple `GET` request with automatic user agent and accept headers. To make more complex requests, you modify the request object with functions that start with `req_`. For example, you could make it a `HEAD` request, with some query parameters, and a custom user agent: req |> req_url_query(param = "value") |> req_user_agent("My user agent") |> req_method("HEAD") |> req_dry_run() #> HEAD /?param=value HTTP/1.1 #> Host: 127.0.0.1:51981 #> User-Agent: My user agent #> Accept: */* #> Accept-Encoding: deflate, gzip Or you could send some JSON in the body of the request: req |> req_body_json(list(x = 1, y = "a")) |> req_dry_run() #> POST / HTTP/1.1 #> Host: 127.0.0.1:51981 #> User-Agent: httr2/0.2.3.9000 r-curl/5.1.0 libcurl/8.1.2 #> Accept: */* #> Accept-Encoding: deflate, gzip #> Content-Type: application/json #> Content-Length: 15 #> #> {"x":1,"y":"a"} httr2 provides a [wide range of `req_` function](https://httr2.r-lib.org/dev/reference/index.html#requests) to customise the request in common ways; if there’s something you need that httr2 doesn’t support, please [file an issue](https://github.com/r-lib/httr2/issues/new) ! Performing the request and handling the response[](#performing-the-request-and-handling-the-response) ------------------------------------------------------------------------------------------------------ Once you have a request that you are happy with, you can send it to the server with [`req_perform()`](https://httr2.r-lib.org/reference/req_perform.html) : req_json <- req |> req_url_path("/json") resp <- req_json |> req_perform() Performing a request will return a response object (or throw an error, which we’ll talk about next). You can see the basic details of the request by printing it or you can see the raw response with [`resp_raw()`](https://httr2.r-lib.org/reference/resp_raw.html) [3](#fn:3) : resp #> #> GET http://127.0.0.1:51981/json #> Status: 200 OK #> Content-Type: application/json #> Body: In memory (407 bytes) resp |> resp_raw() #> HTTP/1.1 200 OK #> Connection: close #> Date: Tue, 14 Nov 2023 14:41:32 GMT #> Content-Type: application/json #> Content-Length: 407 #> ETag: "de760e6d" #> #> { #> "firstName": "John", #> "lastName": "Smith", #> "isAlive": true, #> "age": 27, #> "address": { #> "streetAddress": "21 2nd Street", #> "city": "New York", #> "state": "NY", #> "postalCode": "10021-3100" #> }, #> "phoneNumbers": [\ #> {\ #> "type": "home",\ #> "number": "212 555-1234"\ #> },\ #> {\ #> "type": "office",\ #> "number": "646 555-4567"\ #> }\ #> ], #> "children": [], #> "spouse": null #> } But generally, you’ll want to use the `resp_` functions to extract parts of the response for further processing. For example, you could parse the JSON body into an R data structure: resp |> resp_body_json() |> str() #> List of 8 #> $ firstName : chr "John" #> $ lastName : chr "Smith" #> $ isAlive : logi TRUE #> $ age : int 27 #> $ address :List of 4 #> ..$ streetAddress: chr "21 2nd Street" #> ..$ city : chr "New York" #> ..$ state : chr "NY" #> ..$ postalCode : chr "10021-3100" #> $ phoneNumbers:List of 2 #> ..$ :List of 2 #> .. ..$ type : chr "home" #> .. ..$ number: chr "212 555-1234" #> ..$ :List of 2 #> .. ..$ type : chr "office" #> .. ..$ number: chr "646 555-4567" #> $ children : list() #> $ spouse : NULL Or get the value of a header: resp |> resp_header("Content-Length") #> [1] "407" Error handling[](#error-handling) ---------------------------------- You can use [`resp_status()`](https://httr2.r-lib.org/reference/resp_status.html) to see the returned status: resp |> resp_status() #> [1] 200 But this will almost always be 200, because httr2 automatically follows redirects (statuses in the 300s) and turns HTTP failures (statuses in the 400s and 500s) into R errors. The following example shows what error handling looks like using an example endpoint that returns a response with the status defined in the URL: req |> req_url_path("/status/404") |> req_perform() #> Error in `req_perform()`: #> ! HTTP 404 Not Found. req |> req_url_path("/status/500") |> req_perform() #> Error in `req_perform()`: #> ! HTTP 500 Internal Server Error. Turning HTTP failures into R errors can make debugging hard, so httr2 provides the [`last_request()`](https://httr2.r-lib.org/reference/last_response.html) and [`last_response()`](https://httr2.r-lib.org/reference/last_response.html) helpers which you can use to figure out what went wrong: last_request() #> #> GET http://127.0.0.1:51981/status/500 #> Body: empty last_response() #> #> GET http://127.0.0.1:51981/status/500 #> Status: 500 Internal Server Error #> Content-Type: text/plain #> Body: None httr2 provides two other tools to customise error handling: * [`req_error()`](https://httr2.r-lib.org/reference/req_error.html) gives you full control over what responses should be turned into R errors, and allows you to add additional information to the error message. * [`req_retry()`](https://httr2.r-lib.org/reference/req_retry.html) helps deal with transient errors, where you need to wait a bit and try again. For example, many APIs are rate limited and will return a 429 status if you have made too many requests. You can learn more about both of these functions in “ [Wrapping APIs](https://httr2.r-lib.org/articles/wrapping-apis.html) ” as they are particularly important when creating an R package (or script) that wraps a web API. Control the request process[](#control-the-request-process) ------------------------------------------------------------ There are a number of other `req_` functions that don’t directly affect the HTTP request but instead control the overall process of submitting a request and handling the response. These include: * [`req_cache()`](https://httr2.r-lib.org/reference/req_cache.html) , which sets up a cache so if repeated requests return the same results, and you can avoid a trip to the server. * [`req_throttle()`](https://httr2.r-lib.org/reference/req_throttle.html) , which automatically adds a small delay before each request so you can avoid hammering a server with many requests. * [`req_progress()`](https://httr2.r-lib.org/reference/req_progress.html) , which adds a progress bar for long downloads or uploads. * [`req_cookie_preserve()`](https://httr2.r-lib.org/reference/req_cookie_preserve.html) , which lets you preserve cookies across requests. Additionally, httr2 provides rich support for authenticating with OAuth, implementing many more OAuth flows than httr. You’ve probably used OAuth a bunch without knowing what it’s called: you use it when you login to a non-Google website using your Google account, when you give your phone access to your twitter account, or when you login to a streaming app on your smart TV. OAuth is a big, complex, topic, and is documented in “ [OAuth](https://httr2.r-lib.org/articles/oauth.html) ". Multiple requests[](#multiple-requests) ---------------------------------------- httr2 includes three functions to perform multiple requests: * [`req_perform_sequential()`](https://httr2.r-lib.org/reference/req_perform_sequential.html) takes a list of requests and performs them one at a time. * [`req_perform_parallel()`](https://httr2.r-lib.org/reference/req_perform_parallel.html) takes a list of requests and performs them in parallel (up to 6 at a time by default). It’s similar to [`req_perform_sequential()`](https://httr2.r-lib.org/reference/req_perform_sequential.html) , but is obviously faster, at the expense of potentially hammering a server. It also has some limitations: most importantly it can’t refresh an expired OAuth token and it doesn’t respect [`req_retry()`](https://httr2.r-lib.org/reference/req_retry.html) or [`req_throttle()`](https://httr2.r-lib.org/reference/req_throttle.html) . * [`req_perform_iterative()`](https://httr2.r-lib.org/reference/req_perform_iterative.html) takes a single request and a callback function to generate the next request from previous response. It’ll keep going until the callback function returns `NULL` or `max_reqs` requests have been performed. This is very useful for paginated APIs that only tell you the URL for the _next_ page. For example, imagine we wanted to download each person from the [Star Wars API](https://swapi.dev) . The URLs have a very consistent structure so we can generate a bunch of them, then create the corresponding requests: urls <- paste0("https://swapi.dev/api/people/", 1:10) reqs <- lapply(urls, request) Now I can perform those requests, collecting a list of responses: resps <- req_perform_sequential(reqs) #> Iterating ■■■■ 10% | ETA: 40s #> Iterating ■■■■■■■ 20% | ETA: 3m #> Iterating ■■■■■■■■■■ 30% | ETA: 2m #> Iterating ■■■■■■■■■■■■■ 40% | ETA: 1m #> Iterating ■■■■■■■■■■■■■■■■ 50% | ETA: 46s #> Iterating ■■■■■■■■■■■■■■■■■■■ 60% | ETA: 33s #> Iterating ■■■■■■■■■■■■■■■■■■■■■■ 70% | ETA: 22s #> Iterating ■■■■■■■■■■■■■■■■■■■■■■■■■ 80% | ETA: 13s #> Iterating ■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 90% | ETA: 6s #> Iterating ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 100% | ETA: 0s These responses contain their data in a JSON body: resps |> _[[1]] |> resp_body_json() |> str() #> List of 16 #> $ name : chr "Luke Skywalker" #> $ height : chr "172" #> $ mass : chr "77" #> $ hair_color: chr "blond" #> $ skin_color: chr "fair" #> $ eye_color : chr "blue" #> $ birth_year: chr "19BBY" #> $ gender : chr "male" #> $ homeworld : chr "https://swapi.dev/api/planets/1/" #> $ films :List of 4 #> ..$ : chr "https://swapi.dev/api/films/1/" #> ..$ : chr "https://swapi.dev/api/films/2/" #> ..$ : chr "https://swapi.dev/api/films/3/" #> ..$ : chr "https://swapi.dev/api/films/6/" #> $ species : list() #> $ vehicles :List of 2 #> ..$ : chr "https://swapi.dev/api/vehicles/14/" #> ..$ : chr "https://swapi.dev/api/vehicles/30/" #> $ starships :List of 2 #> ..$ : chr "https://swapi.dev/api/starships/12/" #> ..$ : chr "https://swapi.dev/api/starships/22/" #> $ created : chr "2014-12-09T13:50:51.644000Z" #> $ edited : chr "2014-12-20T21:17:56.891000Z" #> $ url : chr "https://swapi.dev/api/people/1/" There’s lots of ways to deal with this sort of data (e.g. for loops or functional programming) but to make life easier, httr2 comes with its own helper, [`resps_data()`](https://httr2.r-lib.org/reference/resps_successes.html) . This function takes a callback that retrieves the data for each response, then concatenates all the data into a single object. In this case, we need to wrap [`resp_body_json()`](https://httr2.r-lib.org/reference/resp_body_raw.html) in a list, so we get one list for each person, rather than one list in total: resps |> resps_data(\(resp) list(resp_body_json(resp))) |> _[1:3] |> str(list.len = 10) #> List of 3 #> $ :List of 16 #> ..$ name : chr "Luke Skywalker" #> ..$ height : chr "172" #> ..$ mass : chr "77" #> ..$ hair_color: chr "blond" #> ..$ skin_color: chr "fair" #> ..$ eye_color : chr "blue" #> ..$ birth_year: chr "19BBY" #> ..$ gender : chr "male" #> ..$ homeworld : chr "https://swapi.dev/api/planets/1/" #> ..$ films :List of 4 #> .. ..$ : chr "https://swapi.dev/api/films/1/" #> .. ..$ : chr "https://swapi.dev/api/films/2/" #> .. ..$ : chr "https://swapi.dev/api/films/3/" #> .. ..$ : chr "https://swapi.dev/api/films/6/" #> .. [list output truncated] #> $ :List of 16 #> ..$ name : chr "C-3PO" #> ..$ height : chr "167" #> ..$ mass : chr "75" #> ..$ hair_color: chr "n/a" #> ..$ skin_color: chr "gold" #> ..$ eye_color : chr "yellow" #> ..$ birth_year: chr "112BBY" #> ..$ gender : chr "n/a" #> ..$ homeworld : chr "https://swapi.dev/api/planets/1/" #> ..$ films :List of 6 #> .. ..$ : chr "https://swapi.dev/api/films/1/" #> .. ..$ : chr "https://swapi.dev/api/films/2/" #> .. ..$ : chr "https://swapi.dev/api/films/3/" #> .. ..$ : chr "https://swapi.dev/api/films/4/" #> .. ..$ : chr "https://swapi.dev/api/films/5/" #> .. ..$ : chr "https://swapi.dev/api/films/6/" #> .. [list output truncated] #> $ :List of 16 #> ..$ name : chr "R2-D2" #> ..$ height : chr "96" #> ..$ mass : chr "32" #> ..$ hair_color: chr "n/a" #> ..$ skin_color: chr "white, blue" #> ..$ eye_color : chr "red" #> ..$ birth_year: chr "33BBY" #> ..$ gender : chr "n/a" #> ..$ homeworld : chr "https://swapi.dev/api/planets/8/" #> ..$ films :List of 6 #> .. ..$ : chr "https://swapi.dev/api/films/1/" #> .. ..$ : chr "https://swapi.dev/api/films/2/" #> .. ..$ : chr "https://swapi.dev/api/films/3/" #> .. ..$ : chr "https://swapi.dev/api/films/4/" #> .. ..$ : chr "https://swapi.dev/api/films/5/" #> .. ..$ : chr "https://swapi.dev/api/films/6/" #> .. [list output truncated] Another option would be to convert each response into a data frame or tibble. That’s a little tricky here because of the nested lists that will need to become list-columns[4](#fn:4) , so we’ll avoid that challenge here by focussing on the first nine columns: sw_data <- function(resp) { tibble::as_tibble(resp_body_json(resp)[1:9]) } resps |> resps_data(sw_data) #> # A tibble: 10 × 9 #> name height mass hair_color skin_color eye_color birth_year gender #> #> 1 Luke Skywalker 172 77 blond fair blue 19BBY male #> 2 C-3PO 167 75 n/a gold yellow 112BBY n/a #> 3 R2-D2 96 32 n/a white, bl… red 33BBY n/a #> 4 Darth Vader 202 136 none white yellow 41.9BBY male #> 5 Leia Organa 150 49 brown light brown 19BBY female #> 6 Owen Lars 178 120 brown, gr… light blue 52BBY male #> 7 Beru Whitesun… 165 75 brown light blue 47BBY female #> 8 R5-D4 97 32 n/a white, red red unknown n/a #> 9 Biggs Darklig… 183 84 black light brown 24BBY male #> 10 Obi-Wan Kenobi 182 77 auburn, w… fair blue-gray 57BBY male #> # ℹ 1 more variable: homeworld When you’re performing large numbers of requests, it’s almost inevitable that something will go wrong. By default, all three functions will bubble up errors, causing you to lose all of the work that’s been done so far. You can, however, use the `on_error` argument to change what happens, either ignoring errors, or returning when you hit the first error. This will changes the return value: instead of a list of responses, the list might now also contain error objects. httr2 provides other helpers to work with this object: * [`resps_successes()`](https://httr2.r-lib.org/reference/resps_successes.html) filters the list to find the successful responses. You’ll can then pair this with [`resps_data()`](https://httr2.r-lib.org/reference/resps_successes.html) to get the data from the successful request. * [`resps_failures()`](https://httr2.r-lib.org/reference/resps_successes.html) filters the list to find the failed responses. You’ll can then pair this with [`resps_requests()`](https://httr2.r-lib.org/reference/resps_successes.html) to find the requests that generated them and figure out what went wrong,. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 87 folks who have helped make httr2 possible! [@allenbaron](https://github.com/allenbaron) , [@asadow](https://github.com/asadow) , [@atheriel](https://github.com/atheriel) , [@boshek](https://github.com/boshek) , [@casa-henrym](https://github.com/casa-henrym) , [@cderv](https://github.com/cderv) , [@colmanhumphrey](https://github.com/colmanhumphrey) , [@cstjohn810](https://github.com/cstjohn810) , [@cwang23](https://github.com/cwang23) , [@DavidRLovell](https://github.com/DavidRLovell) , [@DMerch](https://github.com/DMerch) , [@dpprdan](https://github.com/dpprdan) , [@ECOSchulz](https://github.com/ECOSchulz) , [@edavidaja](https://github.com/edavidaja) , [@elipousson](https://github.com/elipousson) , [@emmansh](https://github.com/emmansh) , [@Enchufa2](https://github.com/Enchufa2) , [@ErdaradunGaztea](https://github.com/ErdaradunGaztea) , [@fangzhou-xie](https://github.com/fangzhou-xie) , [@fh-mthomson](https://github.com/fh-mthomson) , [@fkohrt](https://github.com/fkohrt) , [@flahn](https://github.com/flahn) , [@gregleleu](https://github.com/gregleleu) , [@guga31bb](https://github.com/guga31bb) , [@gvelasq](https://github.com/gvelasq) , [@hadley](https://github.com/hadley) , [@hongooi73](https://github.com/hongooi73) , [@howardbaek](https://github.com/howardbaek) , [@jameslairdsmith](https://github.com/jameslairdsmith) , [@JBGruber](https://github.com/JBGruber) , [@jchrom](https://github.com/jchrom) , [@jemus42](https://github.com/jemus42) , [@jennybc](https://github.com/jennybc) , [@jimrothstein](https://github.com/jimrothstein) , [@jjesusfilho](https://github.com/jjesusfilho) , [@jjfantini](https://github.com/jjfantini) , [@jl5000](https://github.com/jl5000) , [@jonthegeek](https://github.com/jonthegeek) , [@JosiahParry](https://github.com/JosiahParry) , [@judith-bourque](https://github.com/judith-bourque) , [@juliasilge](https://github.com/juliasilge) , [@kasperwelbers](https://github.com/kasperwelbers) , [@kelvindso](https://github.com/kelvindso) , [@kieran-mace](https://github.com/kieran-mace) , [@KoderKow](https://github.com/KoderKow) , [@lassehjorthmadsen](https://github.com/lassehjorthmadsen) , [@llrs](https://github.com/llrs) , [@lyndon-bird](https://github.com/lyndon-bird) , [@m-mohr](https://github.com/m-mohr) , [@maelle](https://github.com/maelle) , [@maxheld83](https://github.com/maxheld83) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@michaelgfalk](https://github.com/michaelgfalk) , [@misea](https://github.com/misea) , [@MislavSag](https://github.com/MislavSag) , [@mkoohafkan](https://github.com/mkoohafkan) , [@mmuurr](https://github.com/mmuurr) , [@multimeric](https://github.com/multimeric) , [@nbenn](https://github.com/nbenn) , [@nclsbarreto](https://github.com/nclsbarreto) , [@nealrichardson](https://github.com/nealrichardson) , [@Nelson-Gon](https://github.com/Nelson-Gon) , [@olivroy](https://github.com/olivroy) , [@owenjonesuob](https://github.com/owenjonesuob) , [@paul-carteron](https://github.com/paul-carteron) , [@pbulsink](https://github.com/pbulsink) , [@ramiromagno](https://github.com/ramiromagno) , [@rplati](https://github.com/rplati) , [@rressler](https://github.com/rressler) , [@samterfa](https://github.com/samterfa) , [@schnee](https://github.com/schnee) , [@sckott](https://github.com/sckott) , [@sebastian-c](https://github.com/sebastian-c) , [@selesnow](https://github.com/selesnow) , [@Shaunson26](https://github.com/Shaunson26) , [@SokolovAnatoliy](https://github.com/SokolovAnatoliy) , [@spotrh](https://github.com/spotrh) , [@stefanedwards](https://github.com/stefanedwards) , [@taerwin](https://github.com/taerwin) , [@vanhry](https://github.com/vanhry) , [@wing328](https://github.com/wing328) , [@xinzhuohkust](https://github.com/xinzhuohkust) , [@yogat3ch](https://github.com/yogat3ch) , [@yogesh-bansal](https://github.com/yogesh-bansal) , [@yutannihilation](https://github.com/yutannihilation) , and [@zacdav-db](https://github.com/zacdav-db) . * * * 1. Pronounced “hitter 2”. [↩︎](#fnref:1) 2. Well, technically, it does send the request, just to another test server that returns the request that it received. [↩︎](#fnref:2) 3. This is only an approximation. For example, it only shows the final response if there were redirects, and it automatically uncompresses the body if it was compressed. Nevertheless, it’s still pretty useful. [↩︎](#fnref:3) 4. To turn these into list-columns, you need to wrap each list in another list, something like `is_list <- map_lgl(json, is.list); json[is_list] <- map(json[is_list], list)`. This ensures that each element has length 1, the invariant for a row in a tibble. [↩︎](#fnref:4) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # pak 0.6.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) pak 0.6.0 ========= ![](/blog/2023/09/pak-0-6-0/thumbnail-wd.jpg) Photo by [Pixabay](https://www.pexels.com/photo/blue-white-orange-and-brown-container-van-163726/)   2023/09/05   Gábor Csárdi We’re delighted to announce the release of [pak](https://pak.r-lib.org) 0.6.0. pak helps with the installation of R packages and many related tasks. You can install pak from CRAN with: install.packages("pak") If you use an older R version, or a platform that CRAN does not have binary packages for, it is faster and simpler to install pak from our repository. [See the details in the manual.](https://pak.r-lib.org/reference/install.html) This blog post focuses on the exciting new improvements in the matching and installation of system requirements on Linux systems. You can see a full list of changes in the [release notes](https://github.com/r-lib/pak/releases/tag/v0.6.0) System requirements[](#system-requirements) -------------------------------------------- Many R packages require the installation of external software, otherwise they do not work, or even load. For example, the RPostgres R package requires the PostgreSQL client library, and by default dynamically links to it on Linux systems. This means that you (or the administrators of your system) need to install this library, typically in the form of a system package: `libpq-dev` on Ubuntu and Debian systems, or `postgresql-server-devel` or `postgresql-devel` on Red Hat, Fedora, etc. systems. The good news is that pak now helps you with this: * it looks up the required system packages when installing R packages, * it lets you know if any required system packages are missing from your system, before the installation, and * it installs them automatically, if you are a superuser, or if you can use password-less `sudo` to start a superuser shell. In addition, pak now also has some functions to query system requirements and system packages. Supported platforms[](#supported-platforms) -------------------------------------------- pak 0.6.0 supports the following Linux systems currently: * Ubuntu Linux, * Debian Linux, * Red Hat Enterprise Linux, * SUSE Linux Enterprise, * OpenSUSE, * CentOS, * Rocky Linux, * Fedora Linux. Call [`pak::sysreqs_platforms()`](https://pak.r-lib.org/reference/sysreqs_platforms.html) to query the current list of supported platforms: pak::sysreqs_platforms()[,1:3] #> name os distribution #> 1 Ubuntu Linux linux ubuntu #> 2 Debian Linux linux debian #> 3 CentOS Linux linux centos #> 4 Rocky Linux linux rockylinux #> 5 Red Hat Enterprise Linux linux redhat #> 6 Red Hat Enterprise Linux linux redhat #> 7 Red Hat Enterprise Linux linux redhat #> 8 Fedora Linux linux fedora #> 9 openSUSE Linux linux opensuse #> 10 SUSE Linux Enterprise linux sle Call [`pak::system_r_platform()`](https://pak.r-lib.org/reference/system_r_platform.html) to check if pak has detected your platform correctly, and [`pak::sysreqs_is_supported()`](https://pak.r-lib.org/reference/sysreqs_is_supported.html) to see if it is supported: pak::system_r_platform() #> [1] "x86_64-pc-linux-gnu-ubuntu-22.04" pak::sysreqs_is_supported() #> [1] TRUE R package installation[](#r-package-installation) -------------------------------------------------- If you are using pak as the `root` user, on a supported platform, then during package installation pak will look up the required system packages, and will install the missing ones. Here is an example: pak::pkg_install("RPostgres") #> i Loading metadata databasev Loading metadata database ... done #> #> > Will install 12 packages. #> > Will download 12 packages with unknown size. #> + DBI 1.1.3 [dl] #> + RPostgres 1.4.5 [dl] + x libpq-dev #> + Rcpp 1.0.11 [dl] #> + bit 4.0.5 [dl] #> + bit64 4.0.5 [dl] #> + blob 1.2.4 [dl] #> + generics 0.1.3 [dl] #> + hms 1.1.3 [dl] #> + lubridate 1.9.2 [dl] #> + pkgconfig 2.0.3 [dl] #> + timechange 0.2.0 [dl] #> + withr 2.5.0 [dl] #> > Will install 1 system package: #> + libpq-dev - RPostgres #> i Getting 12 pkgs with unknown sizes #> v Got blob 1.2.4 (x86_64-pc-linux-gnu-ubuntu-22.04) (45.94 kB) #> v Got generics 0.1.3 (x86_64-pc-linux-gnu-ubuntu-22.04) (76.24 kB) #> v Got hms 1.1.3 (x86_64-pc-linux-gnu-ubuntu-22.04) (98.35 kB) #> v Got RPostgres 1.4.5 (x86_64-pc-linux-gnu-ubuntu-22.04) (455.11 kB) #> v Got bit64 4.0.5 (x86_64-pc-linux-gnu-ubuntu-22.04) (475.41 kB) #> v Got pkgconfig 2.0.3 (x86_64-pc-linux-gnu-ubuntu-22.04) (17.58 kB) #> v Got timechange 0.2.0 (x86_64-pc-linux-gnu-ubuntu-22.04) (169.26 kB) #> v Got DBI 1.1.3 (x86_64-pc-linux-gnu-ubuntu-22.04) (759.31 kB) #> v Got withr 2.5.0 (x86_64-pc-linux-gnu-ubuntu-22.04) (228.73 kB) #> v Got bit 4.0.5 (x86_64-pc-linux-gnu-ubuntu-22.04) (1.13 MB) #> v Got lubridate 1.9.2 (x86_64-pc-linux-gnu-ubuntu-22.04) (980.37 kB) #> v Got Rcpp 1.0.11 (x86_64-pc-linux-gnu-ubuntu-22.04) (2.15 MB) #> i Installing system requirements #> i Executing `sh -c apt-get -y update` #> i Executing `sh -c apt-get -y install libpq-dev` #> v Installed DBI 1.1.3 (1.1s) #> v Installed RPostgres 1.4.5 (1.1s) #> v Installed Rcpp 1.0.11 (1.2s) #> v Installed bit 4.0.5 (1.2s) #> v Installed bit64 4.0.5 (126ms) #> v Installed blob 1.2.4 (86ms) #> v Installed generics 0.1.3 (83ms) #> v Installed hms 1.1.3 (59ms) #> v Installed lubridate 1.9.2 (1.1s) #> v Installed pkgconfig 2.0.3 (1.1s) #> v Installed timechange 0.2.0 (63ms) #> v Installed withr 2.5.0 (1.1s) #> v 1 pkg + 16 deps: kept 5, added 12, dld 12 (6.58 MB) [17.1s] ### Running R as a regular user[](#running-r-as-a-regular-user) If you don’t want to use R as the superuser, but you can set up `sudo` without a password, that works as well. pak will detect the password-less `sudo` capability, and use it to install system packages, as needed. If you run R as a regular (not root) user, and password-less `sudo` is not available, then pak will print the system requirements, but it will not try to install or update them. If you are compiling R packages from source, and they need to link to system libraries, then their installation will probably fail, until you install these system packages. If you are installing binary R packages (e.g. from [P3M](https://packagemanager.posit.co/client/#/) ), then the installation typically succeeds, but you won’t be able to load these packages into R, until you install the required system packages. To demonstrate this, let’s remove the system package for the PostgreSQL client library: system("apt-get remove -y libpq5") If now we (re)install the binary RPostgres R package, the installation will succeed, but then [`library()`](https://rdrr.io/r/base/library.html) fails because of the missing system package. (We will fix the broken R package below.) #> i Loading metadata databasev Loading metadata database ... done #> #> > Will install 1 package. #> > Will download 1 package with unknown size. #> + RPostgres 1.4.5 [dl] + x libpq-dev #> x Missing 1 system package. You'll probably need to install it manually: #> + libpq-dev - RPostgres #> i Getting 1 pkg with unknown size #> v Cached copy of RPostgres 1.4.5 (x86_64-pc-linux-gnu-ubuntu-22.04) is the latest build #> v Installed RPostgres 1.4.5 (1.1s) #> v 1 pkg + 16 deps: kept 16, added 1 [5.7s] library(RPostgres) #> Error: package or namespace load failed for 'RPostgres' in dyn.load(file, DLLpath = DLLpath, ...): #> unable to load shared object '/root/R/x86_64-pc-linux-gnu-library/4.3/RPostgres/libs/RPostgres.so': #> libpq.so.5: cannot open shared object file: No such file or directory #> Execution halted Opting out[](#opting-out) -------------------------- If you don’t want pak to install system packages for you, set the `PKG_SYSREQS` environment variable to `false`, or the `pkg.sysreqs` option to `FALSE`. See the complete list of configuration options in the [`config?pak`](https://pak.r-lib.org/reference/pak-config.html) manual page. System requirements queries[](#system-requirements-queries) ------------------------------------------------------------ pak 0.6.0 also has a number of functions to query system requirements and system packages. The [`pak::pkg_sysreqs()`](https://pak.r-lib.org/reference/pkg_sysreqs.html) function is similar to [`pak::pkg_deps()`](https://pak.r-lib.org/reference/pkg_deps.html) but in addition to looking up package dependencies, it also looks up system dependencies, and only reports the latter: pak::pkg_sysreqs(c("curl", "r-lib/xml2", "devtools", "CHRONOS")) #> i Loading metadata databasev Loading metadata database ... done #> -- Install scripts --------------------------------------------- Ubuntu 22.04 -- #> apt-get -y update #> apt-get -y install libcurl4-openssl-dev libssl-dev git make libgit2-dev \ #> zlib1g-dev pandoc libfreetype6-dev libjpeg-dev libpng-dev libtiff-dev \ #> libicu-dev libfontconfig1-dev libfribidi-dev libharfbuzz-dev libxml2-dev \ #> libglpk-dev libgmp3-dev default-jdk #> R CMD javareconf #> R CMD javareconf #> #> -- Packages and their system dependencies -------------------------------------- #> CHRONOS -- default-jdk, pandoc #> credentials -- git #> curl -- libcurl4-openssl-dev, libssl-dev #> fs -- make #> gert -- libgit2-dev #> gitcreds -- git #> httpuv -- make, zlib1g-dev #> igraph -- libglpk-dev, libgmp3-dev, libxml2-dev #> knitr -- pandoc #> openssl -- libssl-dev #> pkgdown -- pandoc #> png -- libpng-dev #> ragg -- libfreetype6-dev, libjpeg-dev, libpng-dev, libtiff-dev #> RCurl -- libcurl4-openssl-dev, make #> remotes -- git #> rJava -- default-jdk, make #> rmarkdown -- pandoc #> sass -- make #> stringi -- libicu-dev #> systemfonts -- libfontconfig1-dev, libfreetype6-dev #> textshaping -- libfreetype6-dev, libfribidi-dev, libharfbuzz-dev #> XML -- libxml2-dev #> xml2 -- libxml2-dev See the manual of [`pak::pkg_sysreqs()`](https://pak.r-lib.org/reference/pkg_sysreqs.html) to learn how to programmatically extract information from its return value. [`pak::sysreqs_check_installed()`](https://pak.r-lib.org/reference/sysreqs_check_installed.html) is a handy function that checks if all system requirements are installed for some or all R packages in your library. This should report our broken RPostgres package: pak::sysreqs_check_installed() #> system package installed required by #> -------------- -- ----------- #> libpq-dev x RPostgres [`pak::sysreqs_fix_installed()`](https://pak.r-lib.org/reference/sysreqs_check_installed.html) goes one step further and also tries to install the missing system requirements: pak::sysreqs_fix_installed() #> i Need to install 1 system package. #> i Installing system requirements #> i Executing `sh -c apt-get -y update` #> i Executing `sh -c apt-get -y install libpq-dev` Now we can load RPostgres again: library(RPostgres) Configuration[](#configuration) -------------------------------- There are several pak configuration options you can use to adjust how system requirements are handled. See the complete list in the [`config?pak`](https://pak.r-lib.org/reference/pak-config.html) manual page. Other related pak functions[](#other-related-pak-functions) ------------------------------------------------------------ * [`pak::sysreqs_db_list()`](https://pak.r-lib.org/reference/sysreqs_db_list.html) , `pak::sysreqs_dbmatch()` and [`pak::sysreqs_db_update()`](https://pak.r-lib.org/reference/sysreqs_db_update.html) list, query and update the built-in system requirements database. * [`pak::sysreqs_list_system_packages()`](https://pak.r-lib.org/reference/sysreqs_list_system_packages.html) lists system packages, including virtual packages and the features they provide. More information[](#more-information) -------------------------------------- * [pak documentation](https://pak.r-lib.org/) * [System requirements manual page](https://pak.r-lib.org/reference/sysreqs.html) * [System requirements database](https://github.com/rstudio/r-system-requirements) Acknowledgements[](#acknowledgements) -------------------------------------- A big thank you to all those who have contributed to pak, or one of its workhorse packages since the v0.5.1 release: [@alexpate30](https://github.com/alexpate30) , [@averissimo](https://github.com/averissimo) , [@ArnaudKunzi](https://github.com/ArnaudKunzi) , [@billdenney](https://github.com/billdenney) , [@Darxor](https://github.com/Darxor) , [@drmowinckels](https://github.com/drmowinckels) , [@Fan-iX](https://github.com/Fan-iX) , [@gongyh](https://github.com/gongyh) , [@hadley](https://github.com/hadley) , [@idavydov](https://github.com/idavydov) , [@jefferis](https://github.com/jefferis) , [@joan-yanqiong](https://github.com/joan-yanqiong) , [@kevinushey](https://github.com/kevinushey) , [@kkmann](https://github.com/kkmann) , [@klmr](https://github.com/klmr) , [@krlmlr](https://github.com/krlmlr) , [@lgaborini](https://github.com/lgaborini) , [@maelle](https://github.com/maelle) , [@maxheld83](https://github.com/maxheld83) , [@maximsmol](https://github.com/maximsmol) , [@michaelmayer2](https://github.com/michaelmayer2) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@olivroy](https://github.com/olivroy) , [@pascalgulikers](https://github.com/pascalgulikers) , [@pawelru](https://github.com/pawelru) , [@royfrancis](https://github.com/royfrancis) , [@tanho63](https://github.com/tanho63) , [@thomasyu888](https://github.com/thomasyu888) , [@vincent-hanlon](https://github.com/vincent-hanlon) , and [@VincentGuyader](https://github.com/VincentGuyader) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # teaching [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Teaching [![](/blog/2023/08/teach-tidyverse-23/thumbnail-sq.jpg)](/blog/2023/08/teach-tidyverse-23/) [Teaching the tidyverse in 2023](/blog/2023/08/teach-tidyverse-23/) [learn](/categories/learn) Mine Çetinkaya-Rundel Recommendations for teaching the tidyverse in 2023, summarizing package updates most relevant for teaching data science with the tidyverse, particularly to new learners. [Read more ...](/blog/2023/08/teach-tidyverse-23/) 2023/08/07 [![](/blog/2021/08/teach-tidyverse-2021/thumbnail-sq.jpg)](/blog/2021/08/teach-tidyverse-2021/) [Teaching the tidyverse in 2021](/blog/2021/08/teach-tidyverse-2021/) [learn](/categories/learn) Mine Çetinkaya-Rundel Recommendations for teaching the tidyverse in 2021, summarizing package updates most relevant for teaching data science with the tidyverse, particularly to new learners. [Read more ...](/blog/2021/08/teach-tidyverse-2021/) 2021/08/31 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # testthat 3.2.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) testthat 3.2.0 ============== ![](/blog/2023/10/testthat-3-2-0/thumbnail-wd.jpg) Photo by [Mika Baumeister](https://unsplash.com/photos/PtabTe6iJ_8)   2023/10/08   [devtools](/tags/devtools/) , [testthat](/tags/testthat/)   Hadley Wickham We’re chuffed to announce the release of [testthat](http://testthat.r-lib.org/) 3.2.0. testthat makes it easy to turn your existing informal tests into formal, automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by almost 9,000 CRAN and Bioconductor packages. You can learn more about unit testing at [https://r-pkgs.org/tests.html](https://r-pkgs.org/tests.html) . You can install it from CRAN with: install.packages("testthat") testthat 3.2.0 includes relatively few new features but there have been nine patch releases since testthat 3.1.0. These patch releases contained a bunch of experiments that we now believe are ready for the world. So this blog post summarises the changes in [3.1.1](https://github.com/r-lib/testthat/releases/tag/v3.1.1) , [3.1.2](https://github.com/r-lib/testthat/releases/tag/v3.1.2) , [3.1.3](https://github.com/r-lib/testthat/releases/tag/v3.1.3) , [3.1.4](https://github.com/r-lib/testthat/releases/tag/v3.1.4) , [3.1.5](https://github.com/r-lib/testthat/releases/tag/v3.1.5) , [3.1.6](https://github.com/r-lib/testthat/releases/tag/v3.1.6) , [3.1.7](https://github.com/r-lib/testthat/releases/tag/v3.1.7) , [3.1.8](https://github.com/r-lib/testthat/releases/tag/v3.1.8) , [3.1.9](https://github.com/r-lib/testthat/releases/tag/v3.1.9) , and [3.1.10](https://github.com/r-lib/testthat/releases/tag/v3.1.10) over the last two years. Here we’ll focus on the biggest news: new expectations, tweaks to the way that error snapshots are reported, support for mocking, a new way to detect if a test has changed global state, and a bunch of smaller UI improvements. library(testthat) Documentation[](#documentation) -------------------------------- The first and most important thing to point out is that the second edition of [R Packages](https://r-pkgs.org) contains updated and much expanded coverage of testing. Coverage of testing is now split up over three chapters: * [Testing basics](https://r-pkgs.org/testing-basics.html) * [Designing your test suite](https://r-pkgs.org/testing-design.html) * [Advanced testing techniques](https://r-pkgs.org/testing-advanced.html) There’s also a new vignette about special files ( [`vignette("special-files")`](https://testthat.r-lib.org/articles/special-files.html) ) which describes the various special files that you find in `tests/testthat` and when you might need to use them. New expectations[](#new-expectations) -------------------------------------- There are a handful of notable new expectations. [`expect_contains()`](https://testthat.r-lib.org/reference/expect_setequal.html) and [`expect_in()`](https://testthat.r-lib.org/reference/expect_setequal.html) work similarly to `expect_true(all(expected %in% object))` or `expect_true(all(object %in% expected))` but give more informative failure messages: fruits <- c("apple", "banana", "pear") expect_contains(fruits, "apple") expect_contains(fruits, "pineapple") #> Error: `fruits` (`actual`) doesn't fully contain all the values in "pineapple" (`expected`). #> * Missing from `actual`: "pineapple" #> * Present in `actual`: "apple", "banana", "pear" x <- c(TRUE, FALSE, TRUE, FALSE) expect_in(x, c(TRUE, FALSE)) x <- c(TRUE, FALSE, TRUE, NA, FALSE) expect_in(x, c(TRUE, FALSE)) #> Error: `x` (`actual`) isn't fully contained within c(TRUE, FALSE) (`expected`). #> * Missing from `expected`: NA #> * Present in `expected`: TRUE, FALSE [`expect_no_error()`](https://testthat.r-lib.org/reference/expect_no_error.html) , [`expect_no_warning()`](https://testthat.r-lib.org/reference/expect_no_error.html) , and [`expect_no_message()`](https://testthat.r-lib.org/reference/expect_no_error.html) make it easier (and clearer) to confirm that code runs without errors, warnings, or messages. The default fails if there is any error/warning/message, but you can optionally supply either the `message` or `class` arguments to confirm the absence of a specific error/warning/message. foo <- function(x) { if (x < 0) { x + "10" } else { x = 20 } } expect_no_error(foo(-10)) #> Error: Expected `foo(-10)` to run without any errors. #> ℹ Actually got a with text: #> non-numeric argument to binary operator # No difference here but will lead to a better failure later # once you've fixed this problem and later introduce a new one expect_no_error(foo(-10), message = "non-numeric argument") #> Error: Expected `foo(-10)` to run without any errors matching pattern 'non-numeric argument'. #> ℹ Actually got a with text: #> non-numeric argument to binary operator Snapshotting changes[](#snapshotting-changes) ---------------------------------------------- `expect_snapshot(error = TRUE)` has a new display of error messages that strives to be closer to what you see interactively. In particular, you’ll no longer see the error class and you will now see the error call. * Old display: Code f() Error baz * New display: Code f() Condition Error in `f()`: ! baz If you have used `expect_snapshot(error = TRUE)` in your package, this means that you will need to re-run and approve your snapshots. We hope this is not too annoying and we believe it is worth it given the more accurate reflection of generated error messages. This will not affect checks on CRAN because, by default, snapshot tests are not run on CRAN. Mocking[](#mocking) -------------------- Mocking[1](#fn:1) is a tool for temporarily replacing the implementation of a function in order to make testing easier. Sometimes when testing a function, one part of it is challenging to run in your test environment (maybe it requires human interaction, a live database connection, or maybe it just takes a long time to run). For example, take the following imaginary function. It has a bunch of straightforward computation that would be easy to test but right in the middle of the function it calls `complicated()` which is hard to test: my_function <- function(x, y, z) { a <- f(x, y) b <- g(y, z) c <- h(a, b) d <- complicated(c) i(d, 1, TRUE) } Mocking allows you to temporarily replace `complicated()` with something simpler, allowing you to test the rest of the function. testthat now supports mocking with [`local_mocked_bindings()`](https://testthat.r-lib.org/reference/local_mocked_bindings.html) , which temporarily replaces the implementation of a function. For example, to test `my_function()` you might write something like this: test_that("my_function() returns expected result", { local_mocked_bindings( complicated = function(x) TRUE ) ... }) testthat has a complicated past with mocking. testthat introduced [`with_mock()`](https://testthat.r-lib.org/reference/with_mock.html) in v0.9 (way back in 2014), but we started discovering problems with the implementation in v2.0.0 (2017) leading to its deprecation in v3.0.0 (2020). A few packages arose to fill the gap (like [mockery](https://github.com/r-lib/mockery) , [mockr](https://krlmlr.github.io/mockr/) , and [mockthat](https://nbenn.github.io/mockthat/) ) but none of their implementations were completely satisfactory. Earlier this year a new approach occurred to me that avoids many of the problems of the previous approaches. This is now implemented in [`with_mocked_bindings()`](https://testthat.r-lib.org/reference/local_mocked_bindings.html) and [`local_mocked_bindings()`](https://testthat.r-lib.org/reference/local_mocked_bindings.html) ; we’ve been using these new functions for a few months now without problems, and it feels like time to announce to the world. State inspector[](#state-inspector) ------------------------------------ In times gone by it was very easy to accidentally change the state of the world in a test: test_that("side-by-side diffs work", { options(width = 20) expect_snapshot( waldo::compare(c("X", letters), c(letters, "X")) ) }) When you look at a single test it’s easy to spot the problem, and switch to a more appropriate way of temporarily changing the options, like [`withr::local_options()`](https://withr.r-lib.org/reference/with_options.html) . But sometimes this mistake crept in a long time ago and is now hiding amongst hundreds or thousands of tests. In earlier versions of testthat, finding tests that accidentally changed the world was painful: the only way was to painstakingly review each test. Now you can use [`set_state_inspector()`](https://testthat.r-lib.org/reference/set_state_inspector.html) to register a function that’s called before and after every test. If the function returns different values, testthat will let you know. You’ll typically do this either in `tests/testhat/setup.R` or an existing helper file. So, for example, to detect if any of your tests have modified options you could use this state inspector: set_state_inspector(function() { list(options = options()) }) Or maybe you’ve seen an `R CMD check` warning that you’ve forgotten to close a connection: set_state_inspector(function() { list(connections = nrow(showConnections())) }) And you can of course combine multiple checks just by returning a more complicated list. UI improvements[](#ui-improvements) ------------------------------------ testthat 3.2.0 includes a bunch of minor user interface improvements that should make day-to-day use of testthat more enjoyable. Some of our favourite highlights are: * Parallel testing now works much better with snapshot tests. (And updates to the processx package means that testthat no longer leaves processes around if you terminate a test process early.) * We use an improved algorithm to find the source reference associated with an expectation/error/warning/skip. We now look for the most recent call (within inside [`test_that()`](https://testthat.r-lib.org/reference/test_that.html) that has known source. This generally gives more specific locations than the previous approach and gives much better locations if an error occurs in an exit handler. * Tracebacks are no longer truncated and we use rlang’s default tree display; this should make it easier to track down problems when testing in non-interactive contexts. * Assuming you have a recent RStudio, test failures are now clickable, taking you to the line where the problem occurred. Similarly, when a snapshot test changes, you can now click that suggested code to run the appropriate [`snapshot_accept()`](https://testthat.r-lib.org/reference/snapshot_accept.html) call. * Skips are now only shown at the end of reporter summaries, not as tests are run. This makes them less intrusive in interactive tests while still allowing you to verify that the correct tests are skipped. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 127 contributors who helped make these last 10 release of testthat happen, whether it be through contributed code or filing issues: [@ALanguillaume](https://github.com/ALanguillaume) , [@alessandroaccettulli](https://github.com/alessandroaccettulli) , [@ambica-aas](https://github.com/ambica-aas) , [@annweideman](https://github.com/annweideman) , [@aronatkins](https://github.com/aronatkins) , [@ashander](https://github.com/ashander) , [@AshesITR](https://github.com/AshesITR) , [@astayleraz](https://github.com/astayleraz) , [@ateucher](https://github.com/ateucher) , [@avraam-inside](https://github.com/avraam-inside) , [@b-steve](https://github.com/b-steve) , [@bersbersbers](https://github.com/bersbersbers) , [@billdenney](https://github.com/billdenney) , [@Bisaloo](https://github.com/Bisaloo) , [@cboettig](https://github.com/cboettig) , [@cderv](https://github.com/cderv) , [@chendaniely](https://github.com/chendaniely) , [@ChrisBeeley](https://github.com/ChrisBeeley) , [@ColinFay](https://github.com/ColinFay) , [@CorradoLanera](https://github.com/CorradoLanera) , [@daattali](https://github.com/daattali) , [@damianooldoni](https://github.com/damianooldoni) , [@DanChaltiel](https://github.com/DanChaltiel) , [@danielinteractive](https://github.com/danielinteractive) , [@DavisVaughan](https://github.com/DavisVaughan) , [@daynefiler](https://github.com/daynefiler) , [@dbdimitrov](https://github.com/dbdimitrov) , [@dcaseykc](https://github.com/dcaseykc) , [@dgkf](https://github.com/dgkf) , [@dhicks](https://github.com/dhicks) , [@dimfalk](https://github.com/dimfalk) , [@dougwyu](https://github.com/dougwyu) , [@dpprdan](https://github.com/dpprdan) , [@dvg-p4](https://github.com/dvg-p4) , [@elong0527](https://github.com/elong0527) , [@Enchufa2](https://github.com/Enchufa2) , [@etiennebacher](https://github.com/etiennebacher) , [@FlippieCoetser](https://github.com/FlippieCoetser) , [@florisvdh](https://github.com/florisvdh) , [@gaborcsardi](https://github.com/gaborcsardi) , [@gareth-j](https://github.com/gareth-j) , [@gavinsimpson](https://github.com/gavinsimpson) , [@ghill-fusion](https://github.com/ghill-fusion) , [@hadley](https://github.com/hadley) , [@heavywatal](https://github.com/heavywatal) , [@hfrick](https://github.com/hfrick) , [@hhau](https://github.com/hhau) , [@hpages](https://github.com/hpages) , [@hsloot](https://github.com/hsloot) , [@hughjonesd](https://github.com/hughjonesd) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@jameslairdsmith](https://github.com/jameslairdsmith) , [@jamieRowen](https://github.com/jamieRowen) , [@jayruffell](https://github.com/jayruffell) , [@JBGruber](https://github.com/JBGruber) , [@jennybc](https://github.com/jennybc) , [@JohnCoene](https://github.com/JohnCoene) , [@jonathanvoelkle](https://github.com/jonathanvoelkle) , [@jonthegeek](https://github.com/jonthegeek) , [@josherrickson](https://github.com/josherrickson) , [@kalaschnik](https://github.com/kalaschnik) , [@kapsner](https://github.com/kapsner) , [@kevinushey](https://github.com/kevinushey) , [@kjytay](https://github.com/kjytay) , [@krivit](https://github.com/krivit) , [@krlmlr](https://github.com/krlmlr) , [@larmarange](https://github.com/larmarange) , [@lionel-](https://github.com/lionel-) , [@llrs](https://github.com/llrs) , [@luma-sb](https://github.com/luma-sb) , [@machow](https://github.com/machow) , [@maciekbanas](https://github.com/maciekbanas) , [@maelle](https://github.com/maelle) , [@majr-red](https://github.com/majr-red) , [@maksymiuks](https://github.com/maksymiuks) , [@mardam](https://github.com/mardam) , [@MarkMc1089](https://github.com/MarkMc1089) , [@markschat](https://github.com/markschat) , [@MatthieuStigler](https://github.com/MatthieuStigler) , [@maurolepore](https://github.com/maurolepore) , [@maxheld83](https://github.com/maxheld83) , [@mbojan](https://github.com/mbojan) , [@mcol](https://github.com/mcol) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@mkb13](https://github.com/mkb13) , [@mkoohafkan](https://github.com/mkoohafkan) , [@MKyhos](https://github.com/MKyhos) , [@moodymudskipper](https://github.com/moodymudskipper) , [@Mosk915](https://github.com/Mosk915) , [@mpjashby](https://github.com/mpjashby) , [@ms609](https://github.com/ms609) , [@mtmorgan](https://github.com/mtmorgan) , [@musvaage](https://github.com/musvaage) , [@nealrichardson](https://github.com/nealrichardson) , [@netique](https://github.com/netique) , [@njtierney](https://github.com/njtierney) , [@olivroy](https://github.com/olivroy) , [@osorensen](https://github.com/osorensen) , [@pbulsink](https://github.com/pbulsink) , [@peterdesmet](https://github.com/peterdesmet) , [@r2evans](https://github.com/r2evans) , [@radbasa](https://github.com/radbasa) , [@remlapmot](https://github.com/remlapmot) , [@rfineman](https://github.com/rfineman) , [@rgayler](https://github.com/rgayler) , [@romainfrancois](https://github.com/romainfrancois) , [@s-fleck](https://github.com/s-fleck) , [@salim-b](https://github.com/salim-b) , [@schloerke](https://github.com/schloerke) , [@sorhawell](https://github.com/sorhawell) , [@StatisMike](https://github.com/StatisMike) , [@StatsMan53](https://github.com/StatsMan53) , [@stela2502](https://github.com/stela2502) , [@stla](https://github.com/stla) , [@t-kalinowski](https://github.com/t-kalinowski) , [@tansaku](https://github.com/tansaku) , [@tomliptrot](https://github.com/tomliptrot) , [@torres-pedro](https://github.com/torres-pedro) , [@wes-brooks](https://github.com/wes-brooks) , [@wfmueller29](https://github.com/wfmueller29) , [@wleoncio](https://github.com/wleoncio) , [@wurli](https://github.com/wurli) , [@yogat3ch](https://github.com/yogat3ch) , [@yuliaUU](https://github.com/yuliaUU) , [@yutannihilation](https://github.com/yutannihilation) , and [@zsigmas](https://github.com/zsigmas) . * * * 1. Think mimicking, like a mockingbird, not making fun of. [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyverse [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyverse [![](/blog/2023/08/teach-tidyverse-23/thumbnail-sq.jpg)](/blog/2023/08/teach-tidyverse-23/) [Teaching the tidyverse in 2023](/blog/2023/08/teach-tidyverse-23/) [learn](/categories/learn) Mine Çetinkaya-Rundel Recommendations for teaching the tidyverse in 2023, summarizing package updates most relevant for teaching data science with the tidyverse, particularly to new learners. [Read more ...](/blog/2023/08/teach-tidyverse-23/) 2023/08/07 [![](/blog/2023/03/tidyverse-2-0-0/thumbnail-sq.jpg)](/blog/2023/03/tidyverse-2-0-0/) [tidyverse 2.0.0](/blog/2023/03/tidyverse-2-0-0/) [package](/categories/package) Hadley Wickham Now including lubridate! [Read more ...](/blog/2023/03/tidyverse-2-0-0/) 2023/03/08 [![](/blog/2022/12/stringr-1-5-0/thumbnail-sq.jpg)](/blog/2022/12/stringr-1-5-0/) [stringr 1.5.0](/blog/2022/12/stringr-1-5-0/) [package](/categories/package) Hadley Wickham It’s been a long three years but a new version of stringr is now on CRAN! This release includes a bunch of small but useful new functions and some increased consistency with the rest of the tidyverse. [Read more ...](/blog/2022/12/stringr-1-5-0/) 2022/12/05 [![](/blog/2021/08/teach-tidyverse-2021/thumbnail-sq.jpg)](/blog/2021/08/teach-tidyverse-2021/) [Teaching the tidyverse in 2021](/blog/2021/08/teach-tidyverse-2021/) [learn](/categories/learn) Mine Çetinkaya-Rundel Recommendations for teaching the tidyverse in 2021, summarizing package updates most relevant for teaching data science with the tidyverse, particularly to new learners. [Read more ...](/blog/2021/08/teach-tidyverse-2021/) 2021/08/31 [![](/blog/2021/02/lifecycle-1-0-0/thumbnail-sq.jpg)](/blog/2021/02/lifecycle-1-0-0/) [lifecycle 1.0.0](/blog/2021/02/lifecycle-1-0-0/) [package](/categories/package) Hadley Wickham The lifecycle package documentation received a major overhaul based on what I learned preparing for my rstudio::global keynote. [Read more ...](/blog/2021/02/lifecycle-1-0-0/) 2021/02/15 [![](/blog/2021/02/reprex-1-0-0/thumbnail-sq.jpg)](/blog/2021/02/reprex-1-0-0/) [reprex 1.0.0](/blog/2021/02/reprex-1-0-0/) [package](/categories/package) Jenny Bryan We’ve never blogged about reprex before, so the release of v1.0.0 seems like a good occasion for it. [Read more ...](/blog/2021/02/reprex-1-0-0/) 2021/02/01 [![](/blog/2020/10/readr-1-4-0/thumbnail-sq.jpg)](/blog/2020/10/readr-1-4-0/) [readr 1.4.0](/blog/2020/10/readr-1-4-0/) [package](/categories/package) Jim Hester The newest release of readr brings improved argument consistency, better messages and more flexible output options. [Read more ...](/blog/2020/10/readr-1-4-0/) 2020/10/06 [![](/blog/2020/06/haven-2-3-0/thumbnail-sq.jpg)](/blog/2020/06/haven-2-3-0/) [haven 2.3.0](/blog/2020/06/haven-2-3-0/) [package](/categories/package) Hadley Wickham haven now uses vctrs which means labelled classes will be preserved in tidyr and dplyr operation. [Read more ...](/blog/2020/06/haven-2-3-0/) 2020/06/01 [![](/blog/2020/03/forcats-0-5-0/thumbnail-sq.jpg)](/blog/2020/03/forcats-0-5-0/) [forcats 0.5.0](/blog/2020/03/forcats-0-5-0/) [package](/categories/package) Mara Averick Announcing the release of forcats 0.5.0 on CRAN. [Read more ...](/blog/2020/03/forcats-0-5-0/) 2020/03/02 [![](/blog/2020/02/glue-strings-and-tidy-eval/glue-strings-and-tidy-eval-sq.jpg)](/blog/2020/02/glue-strings-and-tidy-eval/) [Tidy eval now supports glue strings](/blog/2020/02/glue-strings-and-tidy-eval/) [package](/categories/package) Lionel Henry [Read more ...](/blog/2020/02/glue-strings-and-tidy-eval/) 2020/02/11 * [««](/tags/tidyverse/) * « * [1](/tags/tidyverse/) * [2](/tags/tidyverse/page/2/) * [3](/tags/tidyverse/page/3/) *  …  * [6](/tags/tidyverse/page/6/) * [»](/tags/tidyverse/page/2/) * [»»](/tags/tidyverse/page/6/) Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # New interface to validation splits [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) New interface to validation splits ================================== ![](/blog/2023/08/validation-split-as-3-way-split/thumbnail-wd.jpg) Photo by [Scott Webb](https://unsplash.com/photos/68GdK1Aoc8g)   2023/08/25   [tidymodels](/tags/tidymodels/) , [tune](/tags/tune/) , [rsample](/tags/rsample/)   Hannah Frick We’re chuffed to announce the release of a new interface to validation splits in [rsample](https://rsample.tidymodels.org/) 1.2.0 and [tune](https://tune.tidymodels.org/) 1.1.2. The rsample package makes it easy to create resamples for assessing model performance. The tune package facilitates hyperparameter tuning for the tidymodels packages. You can install the new versions from CRAN with: install.packages(c("rsample", "tune")) This blog post will walk you through how to make a validation split and use it for tuning. You can see a full list of changes in the release notes for [rsample](https://github.com/tidymodels/rsample/releases/tag/v1.2.0) and [tune](https://github.com/tidymodels/tune/releases/tag/v1.1.2) . Let’s start with loading the tidymodels package which will load, among others, both rsample and tune. library(tidymodels) #> ── Attaching packages ────────────────────────────────────── tidymodels 1.1.1 ── #> ✔ broom 1.0.5 ✔ recipes 1.0.7 #> ✔ dials 1.2.0 ✔ rsample 1.2.0 #> ✔ dplyr 1.1.2 ✔ tibble 3.2.1 #> ✔ ggplot2 3.4.3 ✔ tidyr 1.3.0 #> ✔ infer 1.0.4 ✔ tune 1.1.2 #> ✔ modeldata 1.2.0 ✔ workflows 1.1.3 #> ✔ parsnip 1.1.1 ✔ workflowsets 1.0.1 #> ✔ purrr 1.0.2 ✔ yardstick 1.2.0 #> ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ── #> ✖ purrr::discard() masks scales::discard() #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ✖ recipes::step() masks stats::step() #> • Use suppressPackageStartupMessages() to eliminate package startup messages The new functions[](#the-new-functions) ---------------------------------------- You can now make a three-way split of your data instead of doing a sequence of two binary splits. * `initial_validation_split()` with variants `initial_validation_time_split()` and `group_initial_validation_split()` for the initial three-way split * `validation_set()` to create the `rset` for tuning containing the analysis (= training) and assessment (= validation) set * `training()`, `validation()`, and `testing()` for access to the separate subsets * `last_fit()` (and `fit_best()`) now also work on the initial three-way split The new functions in action[](#the-new-functions-in-action) ------------------------------------------------------------ To illustrate how to use the new functions, we’ll replicate an analysis of [childcare cost](https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-09/readme.md) from a [Tidy Tuesday](https://github.com/rfordatascience/tidytuesday) done by Julia Silge in one of her [screencasts](https://juliasilge.com/blog/childcare-costs/) . We are modeling the median weekly price for school-aged kids in childcare centers `mcsa` and are thus removing the other variables containing different variants of median prices (e.g., for different age groups). We are also removing the FIPS code identifying the county as we are including various characteristics of the counties instead of their ID. library(readr) #> #> Attaching package: 'readr' #> The following object is masked from 'package:yardstick': #> #> spec #> The following object is masked from 'package:scales': #> #> col_factor childcare_costs <- read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-05-09/childcare_costs.csv') #> Rows: 34567 Columns: 61 #> ── Column specification ──────────────────────────────────────────────────────── #> Delimiter: "," #> dbl (61): county_fips_code, study_year, unr_16, funr_16, munr_16, unr_20to64... #> #> ℹ Use `spec()` to retrieve the full column specification for this data. #> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message. childcare_costs <- childcare_costs |> select(-matches("^mc_|^mfc")) |> select(-county_fips_code) |> drop_na() glimpse(childcare_costs) #> Rows: 23,593 #> Columns: 53 #> $ study_year 2008, 2009, 2010, 2011, 2012, 2013, 2014, 20… #> $ unr_16 5.42, 5.93, 6.21, 7.55, 8.60, 9.39, 8.50, 7.… #> $ funr_16 4.41, 5.72, 5.57, 8.13, 8.88, 10.31, 9.18, 8… #> $ munr_16 6.32, 6.11, 6.78, 7.03, 8.29, 8.56, 7.95, 6.… #> $ unr_20to64 4.6, 4.8, 5.1, 6.2, 6.7, 7.3, 6.8, 5.9, 4.4,… #> $ funr_20to64 3.5, 4.6, 4.6, 6.3, 6.4, 7.6, 6.8, 6.1, 4.6,… #> $ munr_20to64 5.6, 5.0, 5.6, 6.1, 7.0, 7.0, 6.8, 5.9, 4.3,… #> $ flfpr_20to64 68.9, 70.8, 71.3, 70.2, 70.6, 70.7, 69.9, 68… #> $ flfpr_20to64_under6 66.9, 63.7, 67.0, 66.5, 67.1, 67.5, 65.2, 66… #> $ flfpr_20to64_6to17 79.59, 78.41, 78.15, 77.62, 76.31, 75.91, 75… #> $ flfpr_20to64_under6_6to17 60.81, 59.91, 59.71, 59.31, 58.30, 58.00, 57… #> $ mlfpr_20to64 84.0, 86.2, 85.8, 85.7, 85.7, 85.0, 84.2, 82… #> $ pr_f 8.5, 7.5, 7.5, 7.4, 7.4, 8.3, 9.1, 9.3, 9.4,… #> $ pr_p 11.5, 10.3, 10.6, 10.9, 11.6, 12.1, 12.8, 12… #> $ mhi_2018 58462.55, 60211.71, 61775.80, 60366.88, 5915… #> $ me_2018 32710.60, 34688.16, 34740.84, 34564.32, 3432… #> $ fme_2018 25156.25, 26852.67, 27391.08, 26727.68, 2796… #> $ mme_2018 41436.80, 43865.64, 46155.24, 45333.12, 4427… #> $ total_pop 49744, 49584, 53155, 53944, 54590, 54907, 55… #> $ one_race 98.1, 98.6, 98.5, 98.5, 98.5, 98.6, 98.7, 98… #> $ one_race_w 78.9, 79.1, 79.1, 78.9, 78.9, 78.3, 78.0, 77… #> $ one_race_b 17.7, 17.9, 17.9, 18.1, 18.1, 18.4, 18.6, 18… #> $ one_race_i 0.4, 0.4, 0.3, 0.2, 0.3, 0.3, 0.4, 0.4, 0.4,… #> $ one_race_a 0.4, 0.6, 0.7, 0.7, 0.8, 1.0, 0.9, 1.0, 0.8,… #> $ one_race_h 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1,… #> $ one_race_other 0.7, 0.7, 0.6, 0.5, 0.4, 0.7, 0.7, 0.9, 1.4,… #> $ two_races 1.9, 1.4, 1.5, 1.5, 1.5, 1.4, 1.3, 1.6, 2.0,… #> $ hispanic 1.8, 2.0, 2.3, 2.4, 2.4, 2.5, 2.5, 2.6, 2.6,… #> $ households 18373, 18288, 19718, 19998, 19934, 20071, 20… #> $ h_under6_both_work 1543, 1475, 1569, 1695, 1714, 1532, 1557, 13… #> $ h_under6_f_work 970, 964, 1009, 1060, 938, 880, 1191, 1258, … #> $ h_under6_m_work 22, 16, 16, 106, 120, 161, 159, 211, 109, 10… #> $ h_under6_single_m 995, 1099, 1110, 1030, 1095, 1160, 954, 883,… #> $ h_6to17_both_work 4900, 5028, 5472, 5065, 4608, 4238, 4056, 40… #> $ h_6to17_fwork 1308, 1519, 1541, 1965, 1963, 1978, 2073, 20… #> $ h_6to17_mwork 114, 92, 113, 246, 284, 354, 373, 551, 322, … #> $ h_6to17_single_m 1966, 2305, 2377, 2299, 2644, 2522, 2269, 21… #> $ emp_m 27.40, 29.54, 29.33, 31.17, 32.13, 31.74, 32… #> $ memp_m 24.41, 26.07, 25.94, 26.97, 28.59, 27.44, 28… #> $ femp_m 30.68, 33.40, 33.06, 35.96, 36.09, 36.61, 37… #> $ emp_service 17.06, 15.81, 16.92, 16.18, 16.09, 16.72, 16… #> $ memp_service 15.53, 14.16, 15.09, 14.21, 14.71, 13.92, 13… #> $ femp_service 18.75, 17.64, 18.93, 18.42, 17.63, 19.89, 20… #> $ emp_sales 29.11, 28.75, 29.07, 27.56, 28.39, 27.22, 25… #> $ memp_sales 15.97, 17.51, 17.82, 17.74, 17.79, 17.38, 15… #> $ femp_sales 43.52, 41.25, 41.43, 38.76, 40.26, 38.36, 36… #> $ emp_n 13.21, 11.89, 11.57, 10.72, 9.02, 9.27, 9.38… #> $ memp_n 22.54, 20.30, 19.86, 18.28, 16.03, 16.79, 17… #> $ femp_n 2.99, 2.52, 2.45, 2.09, 1.19, 0.77, 0.58, 0.… #> $ emp_p 13.22, 14.02, 13.11, 14.38, 14.37, 15.04, 16… #> $ memp_p 21.55, 21.96, 21.28, 22.80, 22.88, 24.48, 24… #> $ femp_p 4.07, 5.19, 4.13, 4.77, 4.84, 4.36, 6.07, 7.… #> $ mcsa 80.92, 83.42, 85.92, 88.43, 90.93, 93.43, 95… Even after omitting rows with missing values are we left with 23593 observations. That is plenty to work with! We are likely to get a reliable estimate of the model performance from a validation set without having to fit and evaluate the model multiple times, as with, for example, v-fold cross-validation. We are creating a three-way split of the data into a training, a validation, and a test set with the new `initial_validation_split()` function. We are stratifying based on our outcome `mcsa`. The default of `prop = c(0.6, 0.2)` means that 60% of the data gets allocated to the training set and 20% to the validation set - and the remaining 20% go into the test set. set.seed(123) childcare_split <- childcare_costs |> initial_validation_split(strata = mcsa) childcare_split #> #> <14155/4718/4720/23593> You can access the subsets of the data with the familiar `training()` and `testing()` as well as the new `validation()`: validation(childcare_split) #> # A tibble: 4,718 × 53 #> study_year unr_16 funr_16 munr_16 unr_20to64 funr_20to64 munr_20to64 #> #> 1 2013 9.39 10.3 8.56 7.3 7.6 7 #> 2 2011 13.0 12.4 13.6 13.2 12.4 13.9 #> 3 2008 3.85 4.4 3.43 3.7 3.9 3.6 #> 4 2015 8.31 11.8 5.69 7.8 11.7 4.9 #> 5 2015 7.67 6.92 8.27 7.6 6.7 8.3 #> 6 2016 5.95 6.33 5.66 5.7 5.9 5.5 #> 7 2009 10.7 15.9 7.06 8.7 16.8 2.9 #> 8 2010 11.2 15.2 7.89 10.9 14.7 7.8 #> 9 2013 15.0 17.0 13.4 15.2 18.1 13 #> 10 2014 17.4 16.3 18.2 17.2 17.7 16.9 #> # ℹ 4,708 more rows #> # ℹ 46 more variables: flfpr_20to64 , flfpr_20to64_under6 , #> # flfpr_20to64_6to17 , flfpr_20to64_under6_6to17 , #> # mlfpr_20to64 , pr_f , pr_p , mhi_2018 , me_2018 , #> # fme_2018 , mme_2018 , total_pop , one_race , #> # one_race_w , one_race_b , one_race_i , one_race_a , #> # one_race_h , one_race_other , two_races , hispanic , … You may want to extract the training data to do some exploratory data analysis but here we are going to rely on xgboost to figure out patterns in the data so we can breeze straight to tuning a model. xgb_spec <- boost_tree( trees = 500, min_n = tune(), mtry = tune(), stop_iter = tune(), learn_rate = 0.01 ) |> set_engine("xgboost", validation = 0.2) |> set_mode("regression") xgb_wf <- workflow(mcsa ~ ., xgb_spec) xgb_wf #> ══ Workflow ════════════════════════════════════════════════════════════════════ #> Preprocessor: Formula #> Model: boost_tree() #> #> ── Preprocessor ──────────────────────────────────────────────────────────────── #> mcsa ~ . #> #> ── Model ─────────────────────────────────────────────────────────────────────── #> Boosted Tree Model Specification (regression) #> #> Main Arguments: #> mtry = tune() #> trees = 500 #> min_n = tune() #> learn_rate = 0.01 #> stop_iter = tune() #> #> Engine-Specific Arguments: #> validation = 0.2 #> #> Computational engine: xgboost We give this workflow object with the model specification to `tune_grid()` to try multiple combinations of the hyperparameters we tagged for tuning (`min_n`, `mtry`, and `stop_iter`). During tuning, the model should not have access to the test data, only to the data used to fit the model (the analysis set) and the data used to assess the model (the assessment set). Each pair of analysis and assessment set forms a resample. For 10-fold cross-validation, we’d have 10 resamples. With a validation split, we have just one resample with the training set functioning as the analysis set and the validation set as the assessment set. The tidymodels tuning functions all expect a _set_ of resamples (which can be of size one) and the corresponding objects are of class `rset`. To remove the test data from the initial three-way split and create such an `rset` object for tuning, use `validation_set()`. set.seed(234) childcare_set <- validation_set(childcare_split) childcare_set #> # A tibble: 1 × 2 #> splits id #> #> 1 validation We are going to try 15 different parameter combinations and pick the one with the smallest RMSE. set.seed(234) xgb_res <- tune_grid(xgb_wf, childcare_set, grid = 15) #> i Creating pre-processing data to finalize unknown parameter: mtry #> Warning in `[.tbl_df`(x, is.finite(x <- as.numeric(x))): NAs introduced by coercion\ best_parameters <- select_best(xgb_res, "rmse")\ childcare_wflow <- finalize_workflow(xgb_wf, best_parameters)\ \ `last_fit()` then lets you fit your model on the training data and calculate performance on the test data. If you provide it with a three-way split, you can choose if you want your model to be fitted on the training data only or on the combination of training and validation set. You can specify this with the `add_validation_set` argument.\ \ childcare_fit <- last_fit(childcare_wflow, childcare_split, add_validation_set = TRUE)\ collect_metrics(childcare_fit)\ #> # A tibble: 2 × 4\ #> .metric .estimator .estimate .config \ #> \ #> 1 rmse standard 21.4 Preprocessor1_Model1\ #> 2 rsq standard 0.610 Preprocessor1_Model1\ \ \ This takes you through the important changes for validation sets in the tidymodels framework!\ \ Acknowledgements[](#acknowledgements)\ \ --------------------------------------\ \ Many thanks to the people who contributed since the last releases!\ \ For rsample: [@afrogri37](https://github.com/afrogri37)\ , [@AngelFelizR](https://github.com/AngelFelizR)\ , [@bschneidr](https://github.com/bschneidr)\ , [@erictleung](https://github.com/erictleung)\ , [@exsell-jc](https://github.com/exsell-jc)\ , [@hfrick](https://github.com/hfrick)\ , [@jrosell](https://github.com/jrosell)\ , [@MasterLuke84](https://github.com/MasterLuke84)\ , [@MichaelChirico](https://github.com/MichaelChirico)\ , [@mikemahoney218](https://github.com/mikemahoney218)\ , [@rdavis120](https://github.com/rdavis120)\ , [@sametsoekel](https://github.com/sametsoekel)\ , [@Shafi2016](https://github.com/Shafi2016)\ , [@simonpcouch](https://github.com/simonpcouch)\ , [@topepo](https://github.com/topepo)\ , and [@trevorcampbell](https://github.com/trevorcampbell)\ .\ \ For tune: [@blechturm](https://github.com/blechturm)\ , [@cphaarmeyer](https://github.com/cphaarmeyer)\ , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt)\ , [@forecastingEDs](https://github.com/forecastingEDs)\ , [@hfrick](https://github.com/hfrick)\ , [@kjbeath](https://github.com/kjbeath)\ , [@mikemahoney218](https://github.com/mikemahoney218)\ , [@rdavis120](https://github.com/rdavis120)\ , [@simonpcouch](https://github.com/simonpcouch)\ , and [@topepo](https://github.com/topepo)\ .\ \ Contents\ \ The tidyverse is proudly supported by[](https://posit.co/)\ \ [Privacy policy](https://www.tidyverse.org/google_privacy_policy)\ [](https://github.com/tidyverse)\ [](https://twitter.com/hashtag/rstats) --- # deep-dive [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Deep-Dive [![](/blog/2023/08/webr-0-2-0/thumbnail-sq.jpg)](/blog/2023/08/webr-0-2-0/) [webR 0.2.0 has been released](/blog/2023/08/webr-0-2-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.2.0 has been released. Updates and improvements to the webR REPL app, HTML canvas graphics device, internationalisation, Wasm R packages, Shiny support, and the developer API. [Read more ...](/blog/2023/08/webr-0-2-0/) 2023/08/16 [![](/blog/2023/03/webr-0-1-0/thumbnail-sq.jpg)](/blog/2023/03/webr-0-1-0/) [webR 0.1.0 has been released](/blog/2023/03/webr-0-1-0/) [deep-dive](/categories/deep-dive) George Stagg webR 0.1.0 has been released! Using the magic of WebAssembly, webR allows you to run R code directly within a web browser. [Read more ...](/blog/2023/03/webr-0-1-0/) 2023/03/09 [![](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/thumbnail-sq.jpg)](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) [Make your ggplot2 extension package understand the new linewidth aesthetic](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen The next release of ggplot2 will contain a number of internal improvements and fixes long-time inconsistencies. One of these are the conflation of point size and linewidth into the same aesthetic. This post will go into detail with how you can make your extension package work well with the new linewidth aesthetic. [Read more ...](/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) 2022/08/24 [![](/blog/2022/05/case-weights/thumbnail-sq.jpg)](/blog/2022/05/case-weights/) [Using case weights with tidymodels](/blog/2022/05/case-weights/) [deep-dive](/categories/deep-dive) Max Kuhn Support for case weights is now available across many tidymodels packages. [Read more ...](/blog/2022/05/case-weights/) 2022/05/05 [![](/blog/2022/02/new-graphic-features/thumbnail-sq.jpg)](/blog/2022/02/new-graphic-features/) [ragg, svglite, and the new graphics features](/blog/2022/02/new-graphic-features/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen The graphics engine in R is continually improving, and the ragg and svglite graphic devices follow along in supporting them. This blog post outlines what is now possible, and what might come in the future. [Read more ...](/blog/2022/02/new-graphic-features/) 2022/02/25 [![](/blog/2021/11/survival-analysis-parsnip-adjacent/thumbnail-sq.jpg)](/blog/2021/11/survival-analysis-parsnip-adjacent/) [Survival Analysis in tidymodels](/blog/2021/11/survival-analysis-parsnip-adjacent/) [deep-dive](/categories/deep-dive) Hannah Frick We are working on extending support for survival analysis in tidymodels. We are looking for early adopters to try out the new package called censored and give feedback. [Read more ...](/blog/2021/11/survival-analysis-parsnip-adjacent/) 2021/11/02 [![](/blog/2021/02/modern-text-features/modern-text-features-sq.jpg)](/blog/2021/02/modern-text-features/) [Modern Text Features in R](/blog/2021/02/modern-text-features/) [deep-dive](/categories/deep-dive) Thomas Lin Pedersen ragg has taken a major leap forward in text-rendering capabilities with the latest releases of systemfonts, textshaping, and ragg itself. This post will go into detail with what is now possible and how it compares to the build in devices. [Read more ...](/blog/2021/02/modern-text-features/) 2021/02/06 [![](/blog/2019/09/callr-task-q/callr-task-q-sq.jpg)](/blog/2019/09/callr-task-q/) [Multi Process Task Queue in 100 Lines of R Code](/blog/2019/09/callr-task-q/) [programming](/categories/programming) [deep-dive](/categories/deep-dive) Gábor Csárdi We use `callr::r_session` to implement a worker pool and task queue in 100 lines of R code. [Read more ...](/blog/2019/09/callr-task-q/) 2019/09/09 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # book [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Book [![](/blog/2023/08/data-trail/thumbnail-sq.jpg)](/blog/2023/08/data-trail/) [Solutions for R4DS, 2e with Data Trail](/blog/2023/08/data-trail/) [learn](/categories/learn) Jabir Ghaffar, Davon Person, Mine Çetinkaya-Rundel, Tracy Teal Jabir and Davon from Data Trail worked with us to create the solutions manual for R for Data Science, 2nd edition. This blog post summarizes their experience and shares their reflections from working on this project. [Read more ...](/blog/2023/08/data-trail/) 2023/08/02 [![](/blog/2023/07/r4ds-2e/thumbnail-sq.jpg)](/blog/2023/07/r4ds-2e/) [R for Data Science, 2nd edition](/blog/2023/07/r4ds-2e/) [learn](/categories/learn) Mine Çetinkaya-Rundel The second edition of R for Data Science is out, and it’s a major reworking of the first edition. [Read more ...](/blog/2023/07/r4ds-2e/) 2023/07/11 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # webR 0.2.0 has been released [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) webR 0.2.0 has been released ============================ ![](/blog/2023/08/webr-0-2-0/thumbnail-wd.jpg) Photo by [DALL·E 2](https://openai.com/dall-e-2/)   2023/08/16   [webr](/tags/webr/) , [wasm](/tags/wasm/) , [webassembly](/tags/webassembly/)   George Stagg We’re absolutely thrilled to announce the release of [webR](https://docs.r-wasm.org/webr/v0.2.0/) 0.2.0! This release gathers together many updates and improvements to webR over the last few months, including improvements to the HTML canvas graphics device, support for Cairo-based bitmap graphics, accessibility and internationalisation improvements, additional Wasm R package support (including Shiny), a new webR REPL app, and various updates to the webR developer API. This blog post will take a deep dive through the major breaking changes and new features available in webR 0.2.0. I also plan to record and release a series of companion videos discussing the new release, so keep an eye out if you’re someone who prefers watching and listening over reading long-form articles. I’ll update this post with all the links once they’re available. WebAssembly and webR[](#webassembly-and-webr) ---------------------------------------------- My previous [webR release blog post](https://www.tidyverse.org/blog/2023/03/webr-0-1-0/) goes into detail about what WebAssembly is, why people are excited about it, and how it relates to the R community and ecosystem in general through webR. I would recommend it as a good place to start, if the project is new to you[1](#fn:1) . A short explanation is that WebAssembly (also known as Wasm) allows software that’s normally compiled for a specific computer system to instead run anywhere, including in web browsers. Wasm is the technology that powers [Pyodide](https://pyodide.org) (used by [Shinylive for Python](https://shiny.rstudio.com/py/docs/shinylive.html) ) and webR brings this technology to the R world. Using webR it is possible to run R code directly in a web browser[2](#fn:2) , without the need for the traditional supporting R server to execute the code. Running R code directly in a browser opens the door for many new and exciting uses for R on the web. Applications that I’m personally excited in seeing developed are, * Live and interactive R code and graphics in documents & presentations, * Tactile educational content for R, with examples that can be remixed on-the-fly by learners, * Reproducible statistics through containerisation and notebook-style literate programming. Even in these early days, some of this is already being provided by development of downstream projects such as James Balamuta’s [quarto-webr](https://github.com/coatless/quarto-webr) extension, allowing Quarto users to easily embed interactive R code blocks in their documents. ### Interactive code blocks[](#interactive-code-blocks) One of my favourite demonstrations of what webR can do is interactive code blocks for R code. After a short loading period while the webR binary is downloaded, a **Run code** button will be enabled below. Using examples like this, R code can be immediately edited and executed – feel free to experiment! Click the “Run code” button to see the resulting box plot, change the colour from `mediumseagreen` to `red` and run the code again. Loading webR... It’s easy to see the potential teaching benefit examples like this could bring to educational content or R package documentation. The webR REPL app[](#the-webr-repl-app) ---------------------------------------- WebR can be loaded into a web page to be used as a part of a wider web application, and ships with a demo application that does just that. The webR REPL app[3](#fn:3) provides a simple R environment directly in your web browser. The app can be accessed at [https://webr.r-wasm.org/v0.2.0/](https://webr.r-wasm.org/v0.2.0/) and includes sections for R console input/output, code editing, file management, and graphics device output. With the webR REPL app, a casual user could get up and running with R in seconds, without having to install any software on their machine. It is entirely feasible that they could perform the basics of data science entirely within their web browser! Other than interactive code blocks, like in the example earlier, the webR REPL app is perhaps the first thing that users new to webR will interact with. For this reason, we have spent some time working to improve the technical implementation and user experience of using the app. The app has been completely rewritten in the React web framework, replacing the older jQuery library. This allows for better component code organisation and more rapid development of features and updates. [![A screenshot the webR REPL app. The code to generate a ggplot, along with its output, is shown in the app.](repl.png)](repl.png) ### Code editor[](#code-editor) The app now comes with a tabbed code editor, allowing for easier editing and execution of R code. The editor integrates with the webR virtual filesystem (VFS), meaning that multiple R scripts can be opened, edited, and saved and they will be available to the running Wasm R process. The editor pane is built upon the excellent [CodeMirror](https://codemirror.net) text editor, which provides most of the component’s functionality. CodeMirror provides built-in support for syntax highlighting of R code, which is enabled by default when R source files are displayed. The editor is integrated with the currently running R process and automatic code suggestions are shown as you type, provided by R’s [built in completion generator](https://stat.ethz.ch/R-manual/R-devel/library/utils/html/rcompgen.html) . The suggestions are context sensitive and are aware of package and function names, valid arguments, and even objects that exist in the global environment. [![A screenshot of the editor component showing code completion results. One of the suggestions is a data set available in the global environment.](completion.png)](completion.png) The running Wasm R process is also configured at initialisation to use the editor component as its display [pager mechanism](https://stat.ethz.ch/R-manual/R-devel/library/base/html/file.show.html) . With this configuration in place running commands such as [`?rnorm`](https://rdrr.io/r/stats/Normal.html) in the app automatically opens a new read-only tab in the editor displaying R’s built-in documentation. [![A screenshot of the editor component showing built-in R documentation](documentation.png)](documentation.png) ### Plotting pane[](#plotting-pane) The plotting pane has been updated to take advantage of improvements in webR’s HTML canvas graphics device, set as the default device as part of initialisation. In particular, multiple plots are now supported and older plots can be directly accessed using the previous and next buttons in the plotting toolbar. You can try this out with R’s built in graphics demo, by running `demo(graphics)` and/or `demo(persp)`. [![A screenshot of the plot pane showing a built-in R graphics demo](plotting.png)](plotting.png) ### Files pane[](#files-pane) The files pane has been completely redesigned, removing its dependency on jQuery and instead making use of the [react-accessible-treeview](https://www.npmjs.com/package/react-accessible-treeview) package. As well as a technical improvement, this change means that interacting with the webR filesystem should be more usable to those with web accessibility requirements. We feel it’s important that, where possible, everybody is able to use our software. [![A screenshot of the files pane showing the path /home/web_user/plot_random_numbers.R](files.png)](files.png) Additional buttons have also been added to this pane, allowing users to easily manipulate the virtual file system visible to the running Wasm R process. New files and directories can be created or deleted, and text-based files can be directly opened and modified in the editor pane, removing the need to download, edit and then re-upload files. ### Console pane[](#console-pane) The R console component shown in the lower left portion of the app is powered by the wonderful [xterm.js](https://xtermjs.org) software, which provides a high performance terminal emulator on the web. R output looks at its best when running in this kind of environment, so that [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code) can be used to provide a much smoother console experience incorporating cursor placement, box drawing characters, bold text, terminal colours, and more. [![An example of ANSI escape sequences in R console output while loading the tidyverse package.](term.png)](term.png) An optional accessibility mode is provided by xterm.js so that terminal output is readable by screen reader software, such as [macOS’s VoiceOver](https://support.apple.com/en-gb/guide/voiceover/welcome/mac) . The webR REPL app now enables this mode by default to improve the accessibility of terminal output. HTML Canvas graphics device[](#html-canvas-graphics-device) ------------------------------------------------------------ The webR support package provides a custom [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) graphics device that renders output using the [Web Canvas API](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API) . When the graphics device is used, drawing commands from R are translated into Canvas API calls. The browser renders the graphics and the resulting image data is drawn to a HTML `` element on the page. With the release of webR 0.2.0, we have improved the performance and added new features to the HTML canvas graphics device. ### Performance improvements with `OffscreenCanvas`[](#performance-improvements-with-offscreencanvas) Using the Canvas API to draw graphics in a browser is elegant, but presents a problem. R is running via WebAssembly in a JavaScript [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) thread, but the `` element the plot image data is written to is on the main thread, part of the web page [DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction) . And, unfortunately, JavaScript Web Worker threads have no direct access to the DOM. Previous releases of webR solve this problem in a rather naive way, it simply sends the Canvas API calls to the main thread to be executed there. This leads to a few issues, * Canvas API calls are serialised as text to be sent to the main thread. Sufficiently complex plot text must therefore be quoted and escaped. * Each API call is sent in a separate message. For a complex plot this can be thousands of messages to dispatch and handle. * The messaging is one-way, results of useful methods like [`measureText()`](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText) cannot easily be retrieved. * Parsing and executing the API call on the main thread means using JavaScript’s [`eval()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval) or [`Function()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function) , leading to poor performance. These functions should also be avoided when possible in any case, for security reasons. Solid engineering efforts could be made to improve the situation, e.g. through batching API calls and better encoding, but there is a better way: the [`OffscreenCanvas`](https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas) interface. `OffscreenCanvas` is designed to solve this exact problem of rendering graphics off-screen, such as in a worker thread. With `OffscreenCanvas` the Canvas API calls can all be executed on the worker thread, and only a single message containing the completed image data transferred to the main thread when rendering is complete. It is an efficient and technically satisfying solution, except that when webR 0.1.1 was released `OffscreenCanvas` wasn’t supported by the Safari web browser. Today, on the other hand, `OffscreenCanvas` is [supported](https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas#browser_compatibility) in all major desktop and mobile browsers. Safari has supported it since version 16.4, and so with webR 0.2.0 we have rewritten the [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) graphics device to take full advantage of the `OffscreenCanvas` interface. This has led to a significant performance improvement, particularly when creating plots containing many points. The two videos below show the same plot rendered in webR 0.1.1 and 0.2.0, the difference is not just visible, but an order of magnitude faster. A performance comparison plotting 300000 points in webR 0.1.1 and 0.2.0. A potential downside is that users of less up-to-date browsers without `OffscreenCanvas` support won’t be able to use the [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) graphics device. Such users should instead make use of our additional updates to webR to support the traditional Cairo-based bitmap devices. The [built-in graphics devices section](#built-in-bitmap-graphics-devices) discusses that in more detail. ### Modern text rendering and internationalisation[](#modern-text-rendering-and-internationalisation) With webR 0.1.1, the canvas graphics device had only minimal support for rendering text. The typeface was fixed, the font metrics were estimated with a heuristic, and Unicode characters outside the Basic Latin block often failed to render. It worked most of the time, but it was far from ideal. This area of software engineering is [suprisingly difficult](https://faultlore.com/blah/text-hates-you/) to get right, and even native installations of R can have [serious text rendering issues](https://www.tidyverse.org/blog/2021/02/modern-text-features/) . In comparison, web browser support for text rendering is excellent. Now that we use the `OffscreenCanvas` interface, we too can take advantage of the years of work behind browser’s support for text on the web. The example below demonstrates several of the modern text rendering features now supported by [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) . Loading webR... Any system font available to the web browser can now be used[4](#fn:4) . As well as a nice-to-have, this also provides improved accessibility. For example, there are fonts designed specifically for use by readers with dyslexia and other similar reading barriers[5](#fn:5) that could be used for drawing text in plots. Font metrics are now exact, using [`measureText()`](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText) , rather than estimating the width and height of Latin glyphs using heuristics. This gives more accurate positioning of rendered text and improves the general quality of resulting plots. Support for Unicode, font glyph fallback, complex ligatures, and right-to-left (RTL) text have all been improved. This vastly improves results when rendering text for international users, particularly for non-Latin RTL scripts such as the Arabic and Hebrew text in the example above. Also, colour emoji can now be added to plots. 😃 ### Paths and winding rules[](#paths-and-winding-rules) Additional support for the drawing and filling of paths and polygons, including with different [winding rules](https://oreillymedia.github.io/Using_SVG/extras/ch06-fill-rule.html) , has been added to the webR canvas graphics device. An area where this new functionality makes a world of difference is plotting spatial features and maps. Previously broken R code for plotting maps with the `ggplot2` and `sf` packages now works well with webR 0.2.0. [![A screenshot of R plotting code testing paths with winding settings and map plotting. Output on the left for webR 0.1.1 is broken. Output on the right for webR 0.2.0 works correctly](paths.png)](paths.png) ### Output messages from the canvas graphics device[](#output-messages-from-the-canvas-graphics-device) As a result of the changes to the HTML canvas graphics device, the structure of output messages communicated to the main thread has been redesigned. This is a breaking change and existing webR applications will need to be updated to listen for the new output messaging format. [A Plotting section](https://docs.r-wasm.org/webr/latest/plotting.html) has been added to the webR documentation describing how plotting works with the [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) device, and how to handle the output messages in your own web applications. A `'canvas'` type output message with an `event` property of `'canvasNewPage'` indicates the start of a new plot, { type: 'canvas', data: { event: 'canvasNewPage' } } An output message with an `event` property of `'canvasImage'` indicates that there is some graphics data ready to be drawn, { type: 'canvas', data: { event: 'canvasImage', image: ImageBitmap } } The `image` property in the message data contains a JavaScript [`ImageBitmap`](https://developer.mozilla.org/en-US/docs/Web/API/ImageBitmap) object. This can be drawn to a HTML `` element using the [`drawImage()`](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage) method. Built-in bitmap graphics devices[](#built-in-bitmap-graphics-devices) ---------------------------------------------------------------------- Not all environments where webR could be running support plotting to a HTML `` element. Older browsers may not support the required `OffscreenCanvas` interface, webR might be running server-side in Node.js, or webR might be running more traditional R code or packages that are unaware of the [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) graphics device. For supporting these use cases, with webR 0.2.0 the built-in bitmap graphics devices are now able to be used, writing their output to the webR VFS. This includes the [`png()`](https://rdrr.io/r/grDevices/png.html) , [`bmp()`](https://rdrr.io/r/grDevices/png.html) , [`jpeg()`](https://rdrr.io/r/grDevices/png.html) , [`tiff()`](https://rdrr.io/r/grDevices/png.html) devices, and potentially others implemented using the Cairo graphics library. In the example below, webR is loaded into a JavaScript environment and plotting is done using the built-in [`png()`](https://rdrr.io/r/grDevices/png.html) graphics device. The resulting image is written to the virtual filesystem and its contents can then be obtained using webR’s [`FS`](https://docs.r-wasm.org/webr/latest/api/js/classes/WebR.WebR.html#fs) interface, designed to be similar to [Emscripten’s filesystem API](https://emscripten.org/docs/api_reference/Filesystem-API.html) . import { WebR } from 'webr'; const webR = new WebR(); await webR.init(); await webR.evalRVoid(` png('/tmp/Rplot.png', width = 800, height = 800, res = 144) hist(rnorm(1000)) dev.off() `); const plotImageData = await webR.FS.readFile('/tmp/Rplot.png'); The image data is contained in the `plotImageData` variable as a JavaScript [`UInt8Array`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) . Once obtained from the VFS, the image can be served to the end user as a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) file download, displayed on a web page, or if running webR server-side returned over the network. ### Text rendering and font support[](#text-rendering-and-font-support) As with the [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) improvements described in the previous section, we feel it is important that the built in R graphics devices provides a high level of support for text rendering in webR. Here, however, the approach is different. The built-in graphics devices renders image data entirely within the WebAssembly environment, so we can no longer rely on the web browser for high quality text! The built-in graphics devices are powered by the Cairo graphics library, which can now optionally be compiled for Wasm as part of the webR build process. In addition, when enabled various other libraries are compiled for Wasm to improve the quality of text rendering in Cairo, * [pango](https://pango.gnome.org) * [fribidi](http://fribidi.org) * [harfbuzz](https://harfbuzz.github.io) * [freetype](https://freetype.org) * [fontconfig](https://www.freedesktop.org/wiki/Software/fontconfig/) Public releases of webR distributed via GitHub and CDN will be built with these libraries all enabled and included. #### Font files on the VFS[](#font-files-on-the-vfs) When plotting with the built-in bitmap graphics devices, fonts must be accessible to the Cairo library through the webR VFS. A minimal selection of [Google’s Noto fonts](https://fonts.google.com/noto) are bundled with webR when Cairo graphics is enabled. The fontconfig library is also configured to search the VFS directory `/home/web_user/fonts` for additional fonts. Users who wish to use custom fonts, or alternative writing systems, may do so by uploading font files to this directory. In the case of international scripts or non-Latin Unicode such as emoji, fontconfig will automatically use font fallback to select reasonable fonts containing the required glyphs. png(width = 1200, height = 800, res = 180) plot( rnorm(1000), rnorm(1000), col = rgb(0, 0, 0, 0.5), xlim = c(-5, 5), ylim = c(-5, 5), main = "This is the title 🚀", xlab = "This is the x label", ylab = "This is the y label" ) text(-3.5, 4, "This is English") text(-3.5, -4, "هذا مكتوب باللغة العربية") text(3.5, 4, "これは日本語です") text(3.5, -4, "זה כתוב בעברית") dev.off() This is essentially the same example as in the previous section, demonstrating a selection of advanced font functionality. In this example we are rendering a PNG file using the built-in [`png()`](https://rdrr.io/r/grDevices/png.html) graphics device. We can see that by uploading appropriate fonts to the VFS, the same set of advanced text rendering features that are provided by the browser can also be used with R’s built-in bitmap graphics devices. [![A screenshot showing the output of the above plotting code is shown on the left. The additional fonts uploaded to the VFS are listed on the right.](textplot.png)](textplot.png) Lazy virtual filesystem[](#lazy-virtual-filesystem) ---------------------------------------------------- All of the additional features I’ve written about so far come with a price: increased Wasm binary and data download size. Consider the fonts in the previous section - each font file bundled with webR is going to increase the total size of the default webR filesystem by around 500KB. This is a high price to pay in time and bandwidth when not every user is going to need every feature. A similar principle also applies to other files included with R by default. It’s nice that all the default R documentation, examples, and datasets are available on the VFS, but we don’t necessarily need those files downloaded every time to every client machine. With webR 0.2.0 a “lazy” virtual filesystem mechanism, powered by [a feature of Emscripten’s FS API](https://emscripten.org/docs/porting/files/Synchronous-Virtual-XHR-Backed-File-System-Usage.html) , is introduced. With this, only the files required to launch R and use the default packages are downloaded at initialisation time. Additional files provided on the VFS are still available for use, but they are only downloaded from the remote server when they are requested in some way by the running Wasm R process. With the introduction of the lazy virtual filesystem, along with other efficiency improvements, the initial download size for webR is now much smaller, a great improvement. | Component | 0.1.1 | 0.2.0 | (% of previous) | | --- | --- | --- | --- | | `R.bin.data` | 25.3MB | 5.2MB | 20.6% | | `R.bin.wasm` | 12.8MB | 1.7MB | 7.5% | | **Total for the webR REPL app** | 40.2MB | 9.5MB | 23.6% | R packages[](#r-packages) -------------------------- Since initial release, webR has supported loading R packages by first installing them to the Emscripten VFS using the helper function [`webr::install()`](https://docs.r-wasm.org/webr/latest/api/r.html#install-one-or-more-packages-from-a-webr-binary-package-repo) or by manually placing R packages in the VFS at `/usr/lib/R/library`. We find that pure R packages usually work well, but R packages with underlying C (or Fortran, or otherwise…) code must be compiled from source for Wasm. We host a public CRAN-like R package repository containing packages built for Wasm in this way, so that there exists a subset of useful and supported R packages that can be used with webR. The public repository is hosted at [https://repo.r-wasm.org](https://repo.r-wasm.org) and this repo URL is used by default when running [`webr::install()`](https://docs.r-wasm.org/webr/latest/api/r.html#install-one-or-more-packages-from-a-webr-binary-package-repo) to install a Wasm R package. It remains the case that building custom R packages for Wasm is not well documented, but we do hope to improve the situation over time as our package build infrastructure develops and matures. In the future, we plan to provide a Wasm R package build system as a set of Docker containers, so that users are able to build their own packages for webR using a container environment. ### WebAssembly system libraries for R packages[](#webassembly-system-libraries-for-r-packages) Many R packages require linking with system libraries to build and run. When building such R packages for WebAssembly, not only does the package code require compiling for Wasm, but also any system libraries that code depends on. To expand support for R packages, webR 0.2.0 ships with [additional recipes](https://github.com/r-wasm/webr/tree/main/libs/recipes) to build system libraries from source for Wasm. The libraries consist of a selection of utility, database, graphics, text rendering, geometry, and geospatial support packages, with specific libraries chosen for their possibility to be compiled for Wasm as well as the number of R packages relying on them. I expect that the number of system libraries supported will continue to grow over time as we attempt to build more R packages for Wasm. As of webR 0.1.1, **219** packages were available to install through our public Wasm R package repo. With the release of webR 0.2.0 and its additional system libraries, the number of available packages is now **10324** (approximately 51% of CRAN packages). Though, it should be noted that these packages have not been tested in detail. Here, “available” just means that the Emscripten compiler successfully built the R package for Wasm, along with its prerequisite packages. ### Public Wasm R packages dashboard[](#public-wasm-r-packages-dashboard) While available R packages can be listed using [`available.packages()`](https://rdrr.io/r/utils/available.packages.html) with our CRAN-like Wasm R package repo, it’s not the smoothest experience for users simply wanting to check if a given package is available. A dashboard has been added to the [repo index page](https://repo.r-wasm.org) which lists the available packages compiled for Wasm in an interactive table. The table also lists package dependencies, noting which prerequisite packages, if any, are still missing. [![A screenshot of the webR binary R package repository index page. A table of available R packages is shown, along with their prerequisites](repo.png)](repo.png) It might be interesting to note that this dashboard itself is running under webR, through a fully client-side Shiny app. Running httpuv & Shiny under webR[](#running-httpuv--shiny-under-webr) ----------------------------------------------------------------------- Using features new to webR 0.2.0, a [httpuv webR package shim](https://github.com/r-wasm/httpuv) has been created that provides the functionality usually provided by the [httpuv](https://cran.r-project.org/web/packages/httpuv/index.html) R package. The package enables R to handle HTTP and WebSocket traffic, and is a prerequisite for the R Shiny package. The shim works by taking advantage of the [JavaScript Service Worker API](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) . Normally Service Workers are used to implement fast offline caching of web content, but they can also be used as a general network proxy. The httpuv shim makes use of a Service Worker to intercept network traffic from a running Shiny web client, and forward that traffic to be handled by an instance of webR. From the Shiny server’s point of view, it is communicating with the usual httpuv package using its R API. From the point of view of the Shiny web client, it is talking to a Shiny server over the network. Between the two, the JavaScript Service Worker and webR work together to act as a network proxy and handle the traffic entirely within the client[6](#fn:6) . [![A block diagram showing how the httpuv shim, webR worker thread, and Shiny work together. See the preceding diagram for an explanation of how the blocks interact](httpuv.png)](httpuv.png) The httpuv shim package is still in the experimental stage, but it is currently available for testing and is included in our public webR package repository. ### An example shiny app[](#an-example-shiny-app) [![A screenshot of the webR Shiny demo. The shiny app is shown in the top section of the screenshot, an input slider and an output histogram plot. The lower section shows the normally server-side Shiny package console output when tracing is enabled.](shiny.png)](shiny.png) An example Shiny app, making use of the httpuv shim and running fully client-side, is available at [https://shiny-standalone-webr-demo.netlify.app](https://shiny-standalone-webr-demo.netlify.app) . Once the app has loaded in your browser, it’s possible to confirm that the app is running entirely client-side by observing the Shiny server trace output at the bottom of the screen. You should even be able to disconnect completely from the internet and continue to use the app offline. The source code for the demo, which includes some information describing how to set up a webR Shiny server in this way, can be found at [georgestagg/shiny-standalone-webr-demo](https://github.com/georgestagg/shiny-standalone-webr-demo) . Note that this repository is targeted towards advanced web developers with prior experience of development with JavaScript Web Workers. It is intended as a demonstration of the technology, rather than a tutorial. A coming-soon version of Shinylive for R will provide a much better user experience for getting fully client-side R Shiny apps up and running, without requiring advanced knowledge of JavaScript’s Worker API. I believe Shinylive with webR integration will pave the way for providing a user-friendly method to build and deploy containerised R Shiny apps, running on WebAssembly. Changes to the webR developer API[](#changes-to-the-webr-developer-api) ------------------------------------------------------------------------ It’s possible for webR to be used in isolation, but it’s likely that developers will want to interface webR with other JavaScript frameworks and tools. The dynamism and interconnectivity of the web is one of its great strengths, and we’d like the same to be true of webR. This section describes changes to webR’s developer API, used to interact with the running R session from the JavaScript environment. ### Performance improvements with MessagePack protocol[](#performance-improvements-with-messagepack-protocol) When working to integrate webR into a wider application, at some point we will need to move data into the running R process, and later return results back to JavaScript. It’s possible to move data into R by evaluating R code directly, but the webR library also provides [other ways to transfer raw data to R](https://docs.r-wasm.org/webr/latest/convert-js-to-r.html) . Consider the example below. Data is transferred from JavaScript into the running R process by binding `jsData` to an R variable in the global environment using [`webR.objs.globalEnv.bind()`](https://docs.r-wasm.org/webr/latest/convert-js-to-r.html#binding-objects-to-an-r-environment) . Next, some computation on the data is done, represented as evaluating the `do_analysis()` R function. Finally the result is returned back to JavaScript, first as a reference to an R object and then transferring the result data back to the JavaScript environment using [`toJs()`](https://docs.r-wasm.org/webr/latest/convert-r-to-js.html#serialising-r-objects) . const jsData = [... some large JavaScript dataset ...]; await webR.objs.globalEnv.bind('data', jsData); const ret = await webR.evalR("do_analysis(data)"); const result = await ret.toJs(); It’s easy to see how this workflow could be useful as part of a wider application, enabling a complex data manipulation or a statistical modelling in R that would otherwise be awkward to perform directly in JavaScript. Behind the scenes, we’ve done work to ensure that data is transferred efficiently to and from the R environment, and in webR 0.2.0 the [MessagePack](https://msgpack.org/) protocol is now used as the main way that data is serialised and transferred, replacing JSON encoding. This change provides a significant performance improvement. [Initial testing](https://github.com/r-wasm/webr/pull/204) shows an order of magnitude speed boost when transferring large sets of data from the JavaScript environment into R. Thanks to [@jeroen](https://github.com/r-wasm/webr/issues/203) for prompting me to look into it! ### The typing of R object references[](#the-typing-of-r-object-references) When working with webR in TypeScript it is important to keep track of R object types. All references to R objects are instances of the [`RObject`](https://docs.r-wasm.org/webr/latest/objects.html) class, and various subclasses implement specific features for each fundamental R data type. In this example, an [`RDouble`](https://docs.r-wasm.org/webr/latest/api/js/classes/RWorker.RDouble.html) object is returned at runtime, but `webR.evalR()` is typed to return a generic `RObject`. Notice that the [`.toNumber()`](https://docs.r-wasm.org/webr/latest/api/js/classes/RWorker.RDouble.html#tonumber) method exists on `RDouble`, but not on the `RObject` superclass. So while this example runs with no problem once compiled to JavaScript, it gives an error under TypeScript! const obj = await webR.evalR('1.23456'); const data = await obj.toJs(); const num = await obj.toNumber(); // An error under TypeScript! One solution is to use the [`as`](https://www.typescriptlang.org/docs/handbook/2/everyday-types.html#type-assertions) keyword to assert a specific type of `RObject` subclass. Alternatively, webR also provides [variants of the `evalR()` function](https://docs.r-wasm.org/webr/latest/evaluating.html#returning-javascript-values-when-evaluating-r-code) that return and convert results to a specific type of JavaScript object. In many cases these methods will work well, _but they require you to know for sure what type of R object has been returned_. Additional support has been added in webR 0.2.0 to better handle typing when it is not entirely clear what type of `RObject` you have. #### Type predicate functions[](#type-predicate-functions) TypeScript supports a kind of return type known as a [type predicate](https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates) . These return types can be used to create user-defined type guards, functions that take an object argument and return a boolean indicating if the object is of a compatible type. With this, TypeScript is able to automatically [narrow](https://www.typescriptlang.org/docs/handbook/2/narrowing.html) types based on the return value from the type predicate function. WebR 0.2.0 ships with a selection of [type predicate functions for each fundamental R data type](https://docs.r-wasm.org/webr/latest/objects.html#type-predicate-functions) supported by webR. In the following example, the TypeScript error described above is dealt with by using the function [`isRDouble()`](https://docs.r-wasm.org/webr/latest/api/js/modules/RMain.html#isrdouble) . Inside the branch, TypeScript narrows the object type to an `RDouble`, resolving the issue. import { isRDouble } from 'webr'; const obj = await webR.evalR('1.23456'); try { if (isRDouble(obj)) { // In this branch, TypeScript narrows the type of `obj` to an `RDouble` const num = await obj.toNumber(); // Do something with `num` ... } } finally { webR.destroy(obj); } ### Handling errors with `WebRError`[](#handling-errors-with-webrerror) When executing R code with webR’s `evalR()` family of functions, by default any error condition from R is converted into a JavaScript [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) and thrown. This feature can be very useful, because it allows developers to catch issues while executing R code in the native JavaScript environment. However, consider the following example, try { const result = await webR.evalR('some_R_code()'); doSomethingWith(result); } catch (e) { // Handle some error that occured } If an error is thrown, how can we tell if the error came from R or from some issue inside the JavaScript function? Nested `try`/`catch` could be used, but this becomes unwieldy quickly. Parsing the error message text is another option, though not so elegant. With webR 0.2.0 any errors that occur in R code executed using `evalR()`, or any internal webR issues, are thrown as instances of [`WebRError`](https://docs.r-wasm.org/webr/latest/api/js/classes/WebR.WebRError.html) . With this change, the [`instanceof`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/instanceof) keyword can be used to differentiate between errors occurring in R, and errors in JavaScript code. import { WebRError } from 'webR'; try { const result = await webR.evalR('some_R_code()'); doSomethingWith(result); } catch (e) { if (e instanceof WebRError) { console.error("An error occured executing R code"); } else { console.error("An error occured in JavaScript"); } throw e; } ### Safely handling webR termination[](#safely-handling-webr-termination) Consider the following `async` loop, a useful pattern to continuously handle webR output messages, async function run() { for (;;) { const output = await webR.read(); switch (output.type) { case 'stdout': case 'stderr': console.log(output.data); break; default: console.warn(`Unhandled output type: ${output.type}.`); } } } Here `await webR.read()` waits asynchronously for output messages from webR’s communication channel. For example, a running R process might print results between long computational delays. Such occasional printed output might be received as messages with a `type` property of `'stdout'`. After a message is received, it is handled in a `switch` statement and then the loop continues around to wait for another output message. This works well while webR is running, but what happens when terminated with [`webR.close()`](https://docs.r-wasm.org/webr/latest/api/js/classes/WebR.WebR.html#close) ? The R worker thread is stopped and destroyed, but the loop continues to wait for a message that will never come. With webR 0.2.0 a new type of message is issued when webR is terminated using `webR.close()`. After the webR worker thread has been destroyed, a message is emitted on the usual output channel with a `type` property of `'closed'`, with no associated `data` property. The implication is that once this message has been emitted, that particular instance of webR has terminated and the the async loop is no longer needed. With this change, exiting the loop once webR has terminated could be as simple as adding an extra `case` statement, async function run() { for (;;) { const output = await webR.read(); switch (output.type) { case 'stdout': case 'stderr': console.log(output.data); break; case 'closed': return; default: console.warn(`Unhandled output type: ${output.type}.`); } } } Installation and next steps[](#installation-and-next-steps) ------------------------------------------------------------ Developers can integrate webR in their own JavaScript or TypeScript projects by installing the [webR npm package](https://www.npmjs.com/package/webr) , or by directly importing webR from CDN. Issues and PRs are accepted and welcome on the main [r-wasm/webr](https://github.com/r-wasm/webr) GitHub repository. ### npm[](#npm) With this release, the webR npm package name has been updated, simplified from the original `@r-wasm/webr` package name to simply `webr`. npm i webr The original namespaced package `@r-wasm/webr` will be deprecated, and from v0.2.0 onwards npm will display a message pointing to the new package name. ### CDN URL[](#cdn-url) Alternatively, webR can be imported directly as a module from CDN. import { WebR } from "https://webr.r-wasm.org/v0.2.0/webr.mjs" ### Binary release packages[](#binary-release-packages) Finally, binary webR packages can be downloaded from GitHub on the releases page of the [r-wasm/webr](https://github.com/r-wasm/webr) repo. ### Documentation[](#documentation) The next step of integrating webR into your own software should be to visit the documentation pages, provided at [https://docs.r-wasm.org/webr/v0.2.0/](https://docs.r-wasm.org/webr/v0.2.0/) . My previous [webR release blog post](https://www.tidyverse.org/blog/2023/03/webr-0-1-0/) also briefly explains how to get started, though the docs go into much more detail. Acknowledgements[](#acknowledgements) -------------------------------------- A big thank you to all of webR’s early adopters, experimenting with the system and providing feedback in the form of GitHub Issues and PRs. [@Anurodhyadav](https://github.com/Anurodhyadav) , [@arkraieski](https://github.com/arkraieski) , [@averissimo](https://github.com/averissimo) , [@awconway](https://github.com/awconway) , [@bahadzie](https://github.com/bahadzie) , [@ceciliacsilva](https://github.com/ceciliacsilva) , [@DanielEWeeks](https://github.com/DanielEWeeks) , [@eteitelbaum](https://github.com/eteitelbaum) , [@fortunewalla](https://github.com/fortunewalla) , [@gedw99](https://github.com/gedw99) , [@gwd-at](https://github.com/gwd-at) , [@hatemhosny](https://github.com/hatemhosny) , [@hrbrmstr](https://github.com/hrbrmstr) , [@ivelasq](https://github.com/ivelasq) , [@JeremyPasco](https://github.com/JeremyPasco) , [@jeroen](https://github.com/jeroen) , [@jooyoungseo](https://github.com/jooyoungseo) , [@jpjais](https://github.com/jpjais) , [@kforner](https://github.com/kforner) , [@lauritowal](https://github.com/lauritowal) , [@lionel-](https://github.com/lionel-) , [@matthiasbirkich](https://github.com/matthiasbirkich) , [@neocarto](https://github.com/neocarto) , [@noamross](https://github.com/noamross) , [@Polkas](https://github.com/Polkas) , [@qiushiyan](https://github.com/qiushiyan) , [@ries9112](https://github.com/ries9112) , [@SugarRayLua](https://github.com/SugarRayLua) , [@timelyportfolio](https://github.com/timelyportfolio) , [@WebReflection](https://github.com/WebReflection) , and [@WillemSleegers](https://github.com/WillemSleegers) . * * * 1. In addition, [Danielle Navarro’s webR blog post](https://blog.djnavarro.net/posts/2023-04-09_webr/) is very good and Bob Rudis’s [webR experiments](https://rud.is/webr-experiments/) are well worth exploring, along with his recent [NY R conference talk](https://youtu.be/inpwcTUmBDY) . [↩︎](#fnref:1) 2. Also other JavaScript/Wasm environments, such as Node.js. For example, [ROpenSci’s r-universe](https://ropensci.org/r-universe/) package platform provides download links for datasets contained in R packages, in a variety of formats, [powered by running webR server-side in Node.js](https://fosstodon.org/@jeroenooms/110299179903212170) . [↩︎](#fnref:2) 3. REPL stands for “Read, Eval, Print, Loop”, and is another name for the R console that you’re probably familiar with. The application is named the “webR REPL app” because the original version simply provided the user with a fullscreen R console in their web browser. [↩︎](#fnref:3) 4. This also includes the world of CSS web fonts, but it is a little tricky. [Extra work](https://stackoverflow.com/a/53808942) must be done so that the font is available to the Web Worker. Probably this can be handled better in a future release of [`webr::canvas()`](https://docs.r-wasm.org/webr/latest/api/r.html#graphics-device-for-drawing-to-a-html-canvas-element) . [↩︎](#fnref:4) 5. [Dyslexie](https://www.dyslexiefont.com) , [Open Dyslexic](https://opendyslexic.org) . Results of research in this area is mixed, but even if these fonts don’t improve the speed of text comprehension, some users may simply prefer or feel more comfortable with them. [↩︎](#fnref:5) 6. [Shinylive for Python](https://shiny.posit.co/py/docs/shinylive.html) also uses a JavaScript Service Worker scheme to serve fully client-side apps. [↩︎](#fnref:6) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # `purrr::walk()` this way [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) `purrr::walk()` this way ======================== ![](/blog/2023/05/purrr-walk-this-way/thumbnail-wd.jpg) Photo by [Ryoji Iwata](https://unsplash.com/photos/TRJjPc0wss0)   2023/05/26   [fs](/tags/fs/) , [purrr](/tags/purrr/)   Mara Averick Meet the `map()` family[](#meet-the-map-family) ------------------------------------------------ purrr’s [`map()`](https://purrr.tidyverse.org/reference/map.html) family of functions are tools for **iteration**, performing the same action on multiple inputs. If you’re new to purrr, the [Iteration chapter](https://r4ds.had.co.nz/iteration.html#iteration) of R for Data Science is a good place to get started. One of the benefits of using [`map()`](https://purrr.tidyverse.org/reference/map.html) is that the function has variants (e.g.  [`map2()`](https://purrr.tidyverse.org/reference/map2.html) , [`pmap()`](https://purrr.tidyverse.org/reference/pmap.html) , etc.) all of which work the same way. To borrow from Jennifer Thompson’s excellent [Intro to purrr](https://github.com/jenniferthompson/RLadiesIntroToPurrr) ,the arguments can be broken into two groups: what we’re iterating over, and what we’re doing each time. The adapted figure below shows what this looks like for [`map()`](https://purrr.tidyverse.org/reference/map.html) , [`map2()`](https://purrr.tidyverse.org/reference/map2.html) , and [`pmap()`](https://purrr.tidyverse.org/reference/pmap.html) . ![Highlighted titles read: what we're iterating over, and what we're doing each time. For map(.x = , .f = ) .x is what we're iterating over and .f is what we're doing each time. For map2(.x = , .y = , .f = ) .x and .y are what we're iterating over and .f is what we're doing each time. For pmap(.l = list(), .f = ) .l is what we're iterating over and .f is what we're doing each time.](purrr-map-args.png) Grouped map function arguments, adapted from Intro to purrr by Jennifer Thompson In addition to handling different input arguments, the map family of functions has variants that create different outputs. The following table from the [Map-variants section of Advanced R](https://adv-r.hadley.nz/functionals.html#map-variants) shows how the orthogonal inputs and outputs can be used to organise the variants into a matrix: | | List | Atomic | Same type | Nothing | | --- | --- | --- | --- | --- | | One argument | [`map()`](https://purrr.tidyverse.org/reference/map.html) | [`map_lgl()`](https://purrr.tidyverse.org/reference/map.html)
, … | [`modify()`](https://purrr.tidyverse.org/reference/modify.html) | [`walk()`](https://purrr.tidyverse.org/reference/map.html) | | Two arguments | [`map2()`](https://purrr.tidyverse.org/reference/map2.html) | [`map2_lgl()`](https://purrr.tidyverse.org/reference/map2.html)
, … | [`modify2()`](https://purrr.tidyverse.org/reference/modify.html) | [`walk2()`](https://purrr.tidyverse.org/reference/map2.html) | | One argument + index | [`imap()`](https://purrr.tidyverse.org/reference/imap.html) | [`imap_lgl()`](https://purrr.tidyverse.org/reference/imap.html)
, … | [`imodify()`](https://purrr.tidyverse.org/reference/modify.html) | [`iwalk()`](https://purrr.tidyverse.org/reference/imap.html) | | N arguments | [`pmap()`](https://purrr.tidyverse.org/reference/pmap.html) | [`pmap_lgl()`](https://purrr.tidyverse.org/reference/pmap.html)
, … | — | [`pwalk()`](https://purrr.tidyverse.org/reference/pmap.html) | What’s up with `walk()`?[](#whats-up-with-walk) ------------------------------------------------ Based on the table above, you might think that [`walk()`](https://purrr.tidyverse.org/reference/map.html) isn’t very useful. Indeed, [`walk()`](https://purrr.tidyverse.org/reference/map.html) , [`walk2()`](https://purrr.tidyverse.org/reference/map2.html) , and [`pwalk()`](https://purrr.tidyverse.org/reference/pmap.html) all invisibly return `.x`. However, they come in handy when you want to call a function for its _**side effects**_ rather than its return value. Here, we’ll go through two common use cases: saving multiple CSVs, and multiple plots. We’ll also make use of the [fs](https://fs.r-lib.org/) package, a cross-platform interface to file system operations, to inspect our outputs. If you want to try this out but don’t want to save files locally, there’s a [companion project on **Posit Cloud**](https://posit.cloud/content/5983147) where you can follow along. [Test Drive on Posit Cloud](https://posit.cloud/content/5983147) Writing (and deleting) multiple CSVs[](#writing-and-deleting-multiple-csvs) ---------------------------------------------------------------------------- To get started, we’ll need some data. Let’s use the [gapminder](https://googlesheets4.tidyverse.org/reference/gs4_examples.html) example Sheet built into [googlesheets4](https://googlesheets4.tidyverse.org/) . Because there are multiple worksheets (one for each continent), we’ll use [`map()`](https://purrr.tidyverse.org/reference/map.html) to apply [`read_sheet()`](https://googlesheets4.tidyverse.org/reference/range_read.html) [1](#fn:1) to each one, and get back a list of data frames. library(tidyverse) library(googlesheets4) ss <- gs4_example("gapminder") # get sheet id sheets <- sheet_names(ss) # get the names of individual sheets gap_dfs <- map(sheets, .f = \(x) read_sheet(ss, sheet = x)) #> ✔ Reading from gapminder. #> ✔ Range ''Africa''. #> ✔ Reading from gapminder. #> ✔ Range ''Americas''. #> ✔ Reading from gapminder. #> ✔ Range ''Asia''. #> ✔ Reading from gapminder. #> ✔ Range ''Europe''. #> ✔ Reading from gapminder. #> ✔ Range ''Oceania''. Note that the backslash syntax for anonymous functions (e.g. `\(x) x + 1`) was introduced in base R version 4.1.0 as a shorthand for `function(x) x + 1`. If you’re using an earlier version of R, you can use purrr’s shorthand: a formula (e.g. `~ .x + 1`). Typically, you’d want to combine these data frames into one to make it easier to work with your data. To do so, we’ll use [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html) on `gap_dfs`. I’ve kept the intermediary object, since we’ll use it in a moment with [`walk()`](https://purrr.tidyverse.org/reference/map.html) , but could have just as easily piped the output directly. The combination of [`purrr::map()`](https://purrr.tidyverse.org/reference/map.html) and [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html) is a handy one that you can learn more about in the [R for Data Science](https://r4ds.hadley.nz/iteration.html?#purrrmap-and-list_rbind) . gap_combined <- gap_dfs |> list_rbind() Now let’s say that, for whatever reason, you’d like to save the data from these sheets as individual CSVs. This is where [`walk()`](https://purrr.tidyverse.org/reference/map.html) comes into play—writing out the file with [`write_csv()`](https://readr.tidyverse.org/reference/write_delim.html) is a “side effect.” We’ll use [`fs::dir_create()`](https://fs.r-lib.org/reference/create.html) to create a data folder to put our files into[2](#fn:2) , and build a vector of paths/file names. Since we have two arguments, the list of data frames, and the paths, we’ll use [`walk2()`](https://purrr.tidyverse.org/reference/map2.html) . fs::dir_create("data") paths <- str_glue("data/gapminder_{tolower(sheets)}.csv") walk2( gap_dfs, paths, \(df, name) write_csv(df, name) ) To see what we’ve done, we can use [`fs::dir_tree()`](https://fs.r-lib.org/reference/dir_tree.html) to see the contents of the directory as a tree, or [`fs::dir_ls()`](https://fs.r-lib.org/reference/dir_ls.html) to return the paths as a vector. These functions also take `glob` and `regexp` arguments, allowing you to filter paths by file type with globbing patterns (e.g. `*.csv`) or using a regular expression passed on to [`grep()`](https://rdrr.io/r/base/grep.html) . fs::dir_tree("data") #> data #> ├── gapminder_africa.csv #> ├── gapminder_americas.csv #> ├── gapminder_asia.csv #> ├── gapminder_europe.csv #> └── gapminder_oceania.csv fs::dir_ls("data") #> data/gapminder_africa.csv data/gapminder_americas.csv #> data/gapminder_asia.csv data/gapminder_europe.csv #> data/gapminder_oceania.csv If you’re having regrets, or want to return your example project to its previous state, it’s just as easy to [`walk()`](https://purrr.tidyverse.org/reference/map.html) [`fs::file_delete()`](https://fs.r-lib.org/reference/delete.html) along those same paths.[3](#fn:3) walk(paths, \(paths) fs::file_delete(paths)) Saving multiple plots[](#saving-multiple-plots) ------------------------------------------------ Now, let’s say you want to create and save a bunch of plots. We’ll use a modified version of the [`conditional_bars()`](https://r4ds.hadley.nz/functions.html#combining-with-other-tidyverse) [4](#fn:4) function from the R for Data Science chapter on writing [functions](https://r4ds.hadley.nz/functions.html) , and the built-in [diamonds](https://ggplot2.tidyverse.org/reference/diamonds.html) dataset. # modified conditional bars function from R4DS conditional_bars <- function(df, condition, var) { df |> filter({{ condition }}) |> ggplot(aes(x = {{ var }})) + geom_bar() + ggtitle(rlang::englue("Count of diamonds by {{var}} where {{condition}}")) } It’s easy enough to run this for one condition, for example for the diamonds with `cut == "Good"`. diamonds |> conditional_bars(cut == "Good", clarity) ![Bar chart showing count of diamonds by clarity in the diamonds dataset where the cut == Good.](figs/goodclarity-1.png) But what if we want to make and save a plot for each cut? Again, it’s [`map()`](https://purrr.tidyverse.org/reference/map.html) and [`walk()`](https://purrr.tidyverse.org/reference/map.html) to the rescue. Because we’re using the same data (`diamonds`) and conditioning on the same variable (`cut`), we’ll only need to [`map()`](https://purrr.tidyverse.org/reference/map.html) across the levels of `cut`, and can hard code the rest into the anonymous function. # get the levels cuts <- levels(diamonds$cut) # make the plots plots <- map( cuts, \(x) conditional_bars( df = diamonds, cut == {{ x }}, clarity ) ) The plots are now saved in a list—a fine format for storing ggplots. As we did when saving our CSVs, we’ll use fs to create a directory to store them in, and make a vector of paths for file names. # make the folder to put them it (if exists, {fs} does nothing) fs::dir_create("plots") # make the file names plot_paths <- str_glue("plots/{tolower(cuts)}_clarity.png") Now we can use the paths and plots with [`walk2()`](https://purrr.tidyverse.org/reference/map2.html) to pass them as arguments to [`ggsave()`](https://ggplot2.tidyverse.org/reference/ggsave.html) . walk2( plot_paths, plots, \(path, plot) ggsave(path, plot, width = 6, height = 6) ) Again, we can use fs to see what we’ve done: fs::dir_tree("plots") #> plots #> ├── fair_clarity.png #> ├── good_clarity.png #> ├── ideal_clarity.png #> ├── premium_clarity.png #> └── very good_clarity.png And, clean up after ourselves if we didn’t _really_ want those plots after all. walk(plot_paths, \(paths) fs::file_delete(paths)) Fin[](#fin) ------------ Hopefully this gave you a taste for some of what [`walk()`](https://purrr.tidyverse.org/reference/map.html) can do. To learn more, see [Saving multiple outputs](https://r4ds.hadley.nz/iteration.html#saving-multiple-outputs) in the Iteration chapter of R for Data Science. * * * 1. See [Getting started with googlesheets4](https://googlesheets4.tidyverse.org/articles/googlesheets4.html) to learn more about the basics of reading and writing sheets. [↩︎](#fnref:1) 2. If the directory already exists, it will be left unchanged. [↩︎](#fnref:2) 3. There’s also a function in fs called [`dir_walk()`](https://fs.r-lib.org/reference/dir_ls.html) , which you can feel free to explore on your own. [↩︎](#fnref:3) 4. I’ve added a title that reflects the variable name and condition with [`rlang::englue()`](https://rlang.r-lib.org/reference/englue.html) , which you can learn more about in the [Labeling](https://r4ds.hadley.nz/functions.html#labeling) section of the same R4DS chapter. [↩︎](#fnref:4) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # R for Data Science, 2nd edition [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) R for Data Science, 2nd edition =============================== ![](/blog/2023/07/r4ds-2e/thumbnail-wd.jpg) Photo by [Colin Rundel](http://www2.stat.duke.edu/~cr173/)   2023/07/11   [book](/tags/book/)   Mine Çetinkaya-Rundel We’re thrilled to announce the publication of the 2nd edition of [R for Data Science](https://r4ds.hadley.nz/) . The second edition is a major reworking of the first edition, removing material we no longer think is useful, adding material we wish we included in the first edition, and generally updating the text and code to reflect changes in best practices. You can read the book online for free at [](https://r4ds.hadley.nz/) [https://r4ds.hadley.nz](https://r4ds.hadley.nz) , or [buy a physical copy](https://www.amazon.com/dp/1492097403?&tag=hadlwick-20) . Read below to find out what’s new and what’s gone compared to the first edition. What’s new?[](#whats-new) -------------------------- We have renamed the first part of the book to [“Whole game”](https://r4ds.hadley.nz/whole-game.html) , with the goal of giving you the rough details of the "whole game" of data science, including data visualization, transformation, tidying, and import, before we dive into the details. The data visualization chapter has gained a new section written with the [“cake first”](https://datasciencebox.org/01-design-principles.html) approach, which starts with the final visualization you will learn to make, and then builds up to it layer-by-layer. The data tidying chapter introduces the basics of lengthening and widening data and the data import chapter introduces reading tabular data. The second part of the book is ["Visualize"](https://r4ds.hadley.nz/visualize.html) , which gives data visualization tools and best practices a more thorough coverage compared to the first edition. The third part of the book is now called ["Transform"](https://r4ds.hadley.nz/transform.html) and gains new chapters on numbers, logical vectors, and missing values. Much of this content was previously part of the data transformation chapter. In this edition we have expanded them to cover all the details. The fourth part of the book is called ["Import"](https://r4ds.hadley.nz/import.html) , it's a new set of chapters that goes beyond reading flat text files to working with spreadsheets (Excel and GoogleSheets), databases, and big data (with Arrow) as well as rectangling hierarchical data and scraping data from web sites. The ["Program"](https://r4ds.hadley.nz/program.html) part has been rewritten from scratch to focus on the most important parts of function writing and iteration. Function writing now includes details on how to wrap tidyverse functions (dealing with the challenges of tidy evaluation), since this has become much easier and more important over the last few years. We have also added a new chapter on important base R functions that you're likely to see in wild-caught R code. Finally, the ["Communicate"](https://r4ds.hadley.nz/communicate.html) part remains, but has been thoroughly updated to feature [Quarto](https://quarto.org/) instead of R Markdown. This edition of the book has been written in Quarto, and it's clearly the tool of the future. What’s gone?[](#whats-gone) ---------------------------- The first edition of the book featured a part on modeling, which has now been removed. We never had enough room to fully do modelling justice, and there are now much better resources available. We generally recommend using the [tidymodels](https://www.tidymodels.org/) packages and reading [Tidy Modeling with R](https://www.tmwr.org/) by Max Kuhn and Julia Silge. Acknowledgements[](#acknowledgements) -------------------------------------- This book isn't just the product of Hadley, Mine, and Garrett, but is the result of many conversations (in person and online) that we've had with many people in the R community. Huge thanks to [all contributors](https://r4ds.hadley.nz/intro.html#acknowledgments) for the conversations, issues, and pull requests. And, as always, feedback and suggestions are welcome on the [book repository](https://github.com/hadley/r4ds/) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # gmailr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Gmailr [![](/blog/2023/06/gmailr-2-0-0/thumbnail-sq.jpg)](/blog/2023/06/gmailr-2-0-0/) [gmailr 2.0.0](/blog/2023/06/gmailr-2-0-0/) [package](/categories/package) Jennifer Bryan gmailr 2.0.0 streamlines the auth process and makes it easier to use gmailr in a cloud or deployed context. [Read more ...](/blog/2023/06/gmailr-2-0-0/) 2023/06/29 [![](/blog/2019/08/gmailr-1-0-0/gmailr-1-0-0-sq.jpg)](/blog/2019/08/gmailr-1-0-0/) [gmailr v1.0.0](/blog/2019/08/gmailr-1-0-0/) [package](/categories/package) Jim Hester gmailr v1.0.0 is on CRAN. [Read more ...](/blog/2019/08/gmailr-1-0-0/) 2019/08/26 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # gargle [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Gargle [![](/blog/2023/06/gmailr-2-0-0/thumbnail-sq.jpg)](/blog/2023/06/gmailr-2-0-0/) [gmailr 2.0.0](/blog/2023/06/gmailr-2-0-0/) [package](/categories/package) Jennifer Bryan gmailr 2.0.0 streamlines the auth process and makes it easier to use gmailr in a cloud or deployed context. [Read more ...](/blog/2023/06/gmailr-2-0-0/) 2023/06/29 [![](/blog/2021/08/bigrquery-1-4-0/thumbnail-sq.jpg)](/blog/2021/08/bigrquery-1-4-0/) [bigrquery 1.4.0](/blog/2021/08/bigrquery-1-4-0/) [package](/categories/package) Jenny Bryan bigrquery 1.4.0 fixes a bug in `bq_table_download()`. [Read more ...](/blog/2021/08/bigrquery-1-4-0/) 2021/08/04 [![](/blog/2021/07/googlesheets4-1-0-0/thumbnail-sq.jpg)](/blog/2021/07/googlesheets4-1-0-0/) [googlesheets4 1.0.0](/blog/2021/07/googlesheets4-1-0-0/) [package](/categories/package) Jenny Bryan Version 1.0.0 marks the graduation of googlesheets4 from experimental to stable. [Read more ...](/blog/2021/07/googlesheets4-1-0-0/) 2021/07/26 [![](/blog/2021/07/gargle-1-2-0/thumbnail-sq.jpg)](/blog/2021/07/gargle-1-2-0/) [gargle 1.2.0](/blog/2021/07/gargle-1-2-0/) [package](/categories/package) Jenny Bryan gargle has seen a lot of development over the past two years and five releases: cache relocation, credential rolling, a new auth method, an improved user interface, better verbosity control, and retries. [Read more ...](/blog/2021/07/gargle-1-2-0/) 2021/07/04 [![](/blog/2019/08/gargle-hello-world/gargle-hello-world-sq.jpg)](/blog/2019/08/gargle-hello-world/) [gargle’s debut on CRAN](/blog/2019/08/gargle-hello-world/) [package](/categories/package) Jenny Bryan gargle is now on CRAN. [Read more ...](/blog/2019/08/gargle-hello-world/) 2019/08/20 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # spring cleaning [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Spring Cleaning [![](/blog/2023/06/spring-cleaning-2023/thumbnail-sq.jpg)](/blog/2023/06/spring-cleaning-2023/) [Package spring cleaning](/blog/2023/06/spring-cleaning-2023/) [other](/categories/other) Andy Teucher When Spring comes around, it’s time to embark on some Spring Cleaning to take care of the package maintenance tasks that you never seem to get around to. This post outlines the processes and tools that the tidyverse team uses to make this job fun and efficient. [Read more ...](/blog/2023/06/spring-cleaning-2023/) 2023/06/07 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Solutions for R4DS, 2e with Data Trail [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Solutions for R4DS, 2e with Data Trail ====================================== ![](/blog/2023/08/data-trail/thumbnail-wd.jpg) Photo by [Tim Foster](https://unsplash.com/photos/BnwRf_m3EIg)   2023/08/02   [internship](/tags/internship/) , [book](/tags/book/)   Jabir Ghaffar, Davon Person, Mine Çetinkaya-Rundel, Tracy Teal Last year at rstudio::conf(2022) Jeff Leek [shared about the Data Trail program](https://youtu.be/Vf301YCxP1Q) . > DataTrail is a no-cost, paid 14-week educational initiative for young-adult, high school and GED-graduates. DataTrail aims to equip members of underserved communities with the necessary skills and support required to work in the booming field of data science. DataTrail is a fresh take on workforce development that focuses on training both Black, Indigenous, and other people of color (BIPOC) interested in the data science industry _**and**_ their potential employers. We have been so excited then to have the opportunity to work with two Data Trail interns this year! [Jabir Ghaffar](https://jabirghaffar.quarto.pub/jabir/) went through the Data Trail program June 2022, and [Davon Person](https://www.linkedin.com/in/davon-person-1ba973194/) went through the Data Trail program in 2019, and now works as a Data Programming Specialist with the project. Jabir and Davon worked on solutions for the R for Data Science book, explored some Tidy Tuesday datasets, created their own Quarto websites, and their perspectives helped us learn more about how our tools and documentation can better support emerging data scientists. Jabir’s primary project was to work on the [R for Data Science solutions](https://mine-cetinkaya-rundel.github.io/r4ds-solutions/) . The R for Data Science, 2nd Edition was released in June 2023. In this edition there is a lot of new content, and revisions and additions to exercises. We saw that [Jeffrey Arnold’s solutions to the 1st edition](https://jrnold.github.io/r4ds-exercise-solutions/) are such a useful resource for the community. Therefore, this project aimed to both create a similar resource for 2nd edition and serve as an educational resource for the interns to help sharpen their tidyverse and general data science skills. Some learning highlights for Jabir included faceting (as a solution to overplotting), consistent code styling (which helps make the code more pleasing to read), and, making maps! Specifically, Jabir mentioned that he has always wondered “how did they do that?!” with maps and found them to be quite intimidating, so it was especially satisfying to create his first heatmap of US states and chance of getting a tornado ([https://jabirghaffar.quarto.pub/jabir/posts/tornado\_mapping\_exploration](https://jabirghaffar.quarto.pub/jabir/posts/tornado_mapping_exploration) ). This was not just a satisfying visualization exercise, but also a great opportunity to dig into unfamiliar data wrangling functions like [`recode()`](https://dplyr.tidyverse.org/reference/recode.html) for converting 2-letter state abbreviations to state names. The exercises in R4DS range from quick, almost obvious, drills to ones that can really make you think and spin your wheels for a bit. One such exercise for Jabir was the one on changing the display of presidential terms from [Section 12.4.6](https://r4ds.hadley.nz/communication.html#exercises-2) . The exercise asks: > Change the display of the presidential terms by: > > * Combining the two variants that customize colors and x axis breaks. > * Improving the display of the y axis. > * Labelling each term with the name of the president. > * Adding informative plot labels. > * Placing breaks every 4 years (this is trickier than it seems!). The starting points for the exercise are the following plots from the text: library(tidyverse) presidential |> mutate(id = 33 + row_number()) |> ggplot(aes(x = start, y = id)) + geom_point() + geom_segment(aes(xend = end, yend = id)) + scale_x_date(name = NULL, breaks = presidential$start, date_labels = "'%y") presidential |> mutate(id = 33 + row_number()) |> ggplot(aes(x = start, y = id, color = party)) + geom_point() + geom_segment(aes(xend = end, yend = id)) + scale_color_manual(values = c(Republican = "#E81B23", Democratic = "#00AEF3")) ![](figs/presidential-terms-start-1.png)![](figs/presidential-terms-start-2.png) Jabir says “This question completely puzzled me. I’d say good luck, and if you’re new and you get to this question, I recommend you look at the solution manual.” The first challenge was identifying where in the text the original plot was developed, and the code associated with it. And then, the most challenging part of this exercise was labeling the y-axis with the names of presidents. It took lots of Googling, but ultimately Jabir used suggestions from ChatGPT to get this over the finish line. And, perhaps, the frustrating and satisfying part is that the answer was pretty obvious in hindsight: presidential <- presidential |> mutate(id = 33 + row_number()) ggplot(presidential, aes(x = start, y = id)) + geom_point(aes(color = party)) + geom_segment(aes(xend = end, yend = id, color = party)) + geom_text(aes(label = name), hjust = 0, vjust = 0, nudge_y = 0.1) + scale_color_manual(values = c(Republican = "#E81B23", Democratic = "#00AEF3")) + scale_x_date( name = "Term", breaks = seq(from = ymd("1953-01-20"), to = ymd("2021-01-20"), by = "4 years"), date_labels = "'%y" ) + scale_y_continuous(breaks = 34:45) + theme( panel.grid.minor = element_blank(), axis.ticks.y = element_blank() ) + labs( x = "Term", y = "President", title = "Terms of US Presidents", subtitle = "Eisenhower (34th) to Trump (45th)", color = "Party" ) ![](figs/presidential-terms-end-1.png) Jabir felt like moments when he knew what to do and how to answer each question were very satisfying, but the moments where he felt stuck and went into rabbit holes looking for answers made him question whether he wanted to continue becoming a data scientist. Ultimately, though, the project was enjoyable, and not just a great learning experience for Jabir, but also a very meaningful one because it created a resource that can help future data scientists. We felt like Jabir and Davon advanced their data science skills, and familiarity with working in open source, throughout the project. It was particularly exciting to see Jabir create his own data science portfolio as a Quarto website with posts on the Tidy Tuesday datasets, and we really appreciated his work on the R4DS Solutions. In going through those, he helped better refine the book, and created a resource that so many people are going to be able to use and learn from. We could see how he learned not just data science, but also grew as a leader who will continue to support others in their learning as he moves on to work as a developer with the Data Trail program. Davon too focused not just on data science, but was a part of our team, and the Data Trail team, providing mentorship to Jabir and bringing teaching approaches, like using Tidy Tuesday datasets, to his community. We are so grateful to have had the opportunity to work with them both, and see this as the beginning of continued collaborations. Just like most open-source projects, the [R4DS Solutions](https://mine-cetinkaya-rundel.github.io/r4ds-solutions/) is a living and breathing project, still a work-in-progress. We would welcome any community contributions! All perspectives are important here, and it’s a great project if you’re a first-time contributor. The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Tidyteam code review principles [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyteam code review principles =============================== ![](/blog/2023/06/code-review-principles/thumbnail-wd.jpg) Photo by [Lanju Fotografie](https://unsplash.com/photos/BvAoCypqRXU)   2023/06/05   Davis Vaughan At Posit, we strive to write high quality code to ensure that you, our users, have the best experience possible. We feel that the code review process plays a critical role in delivering quality products, and in developing the skills of newer contributors, and we decided to make that process explicit through a [tidyteam code review principles](https://code-review.tidyverse.org/) guide. At a high level, this guide walks you through the perspectives of both the pull request author and the pull request reviewer, discussing various aspects of the process from both points of view (such as how to [handle reviewer comments](https://code-review.tidyverse.org/author/handling-comments.html) and how to write [focused pull requests](https://code-review.tidyverse.org/author/focused.html) ). Throughout the guide, we repeatedly tie back to three different [patterns of collaboration](https://code-review.tidyverse.org/collaboration/) , which reflect that each code review is unique and comes with its own set of expectations between the author and the reviewer. We posted about this guide on [Twitter](https://twitter.com/dvaughan32/status/1645866331487756288?s=20) and [Mastodon](https://fosstodon.org/@davis/110181751636631782) a few weeks ago: > In the tidyverse, we work with a lot of people - each other and [#rstats](https://twitter.com/hashtag/rstats?src=hash&ref_src=twsrc%5Etfw) > community members. > > We wanted to document how we handle code review, so we\\'ve drafted a guide detailing our review principles! > > We hope you find it useful, and welcome your feedback![https://t.co/jGm0rSGg5M](https://t.co/jGm0rSGg5M) > > \--- Davis Vaughan (@dvaughan32) [April 11, 2023](https://twitter.com/dvaughan32/status/1645866331487756288?ref_src=twsrc%5Etfw) And we were happy to see that [many of you](https://fosstodon.org/@jromanowska/110182021271601892) are already finding it useful! In particular, I’d like to shout out [Hiroaki Yutani](https://github.com/yutannihilation) who created a [two part video series](https://www.youtube.com/watch?v=gSv6h2heHQE) reading through the principles in Japanese! Internally, we’ve also been referencing this guide when reviewing pull requests from each other and from the community. For example, Jenny Bryan linked out to the section on [creating a good pull request description](https://code-review.tidyverse.org/author/submitting.html#sec-descriptions) when reviewing a [bigrquery PR](https://github.com/r-dbi/bigrquery/pull/512#issuecomment-1511687647) , and I internally linked a colleague to the section on [GitHub Suggestions](https://code-review.tidyverse.org/reviewer/comments.html#github-suggestions) , which discusses how to batch multiple suggestions into a single commit. We adapted these principles from Google’s own [guide](https://google.github.io/eng-practices/review/) , and we encourage you to do the same thing with ours. If you work in a research lab or are on a software team at your company, then code review should be as important to you as it is to us! Feel free to modify these principles to suit your own needs, and if you do use them, we’d love to hear about it. The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q2 2023 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q2 2023 tidymodels digest ========================= ![](/blog/2023/07/tidymodels-2023-q2/thumbnail-wd.jpg) Photo by [United States Geological Survey](https://unsplash.com/photos/PuLsDCBbyBM)   2023/07/19   [tidymodels](/tags/tidymodels/)   Hannah Frick The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like the [post](https://www.tidyverse.org/blog/2023/05/desirability2/) on the release of the new desirability2 package. Since [our last roundup post](https://www.tidyverse.org/blog/2023/04/tidymodels-2023-q1/) , there have been CRAN releases of 7 tidymodels packages. Here are links to their NEWS files: * agua [(0.1.3)](https://agua.tidymodels.org/news/index.html) * broom [(1.0.5)](https://broom.tidymodels.org/news/index.html) * desirability2 [(0.0.1)](https://desirability2.tidymodels.org/news/index.html) * embed [(1.1.1)](https://embed.tidymodels.org/news/index.html) * probably [(1.0.2)](https://probably.tidymodels.org/news/index.html) * spatialsample [(0.4.0)](https://spatialsample.tidymodels.org/news/index.html) * tidymodels [(1.1.0)](https://tidymodels.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: a new package with data for modeling, nearest neighbor distance matching cross-validation for spatial data, and a website refresh. First, loading the collection of packages: library(tidymodels) modeldatatoo[](#modeldatatoo) ------------------------------ Many of the datasets used in tidymodels examples are available in the modeldata package. The new modeldatatoo package now extends the collection by several bigger datasets. To allow for the bigger size, the package does not contain those datasets directly but rather provides functions to access them, prefixed with `data_`. For example: library(modeldatatoo) data_animals() #> # A tibble: 610 × 48 #> text colour lifespan weight kingdom class phylum diet conservation_status #> #> 1 "Aardv… Brown… 23 years 60kg … Animal… Mamm… Chord… Omni… Least Concern #> 2 "Abyss… Fawn,… NA NA NA NA NA NA NA #> 3 "Adeli… Black… 10 - 20… 3kg -… Animal… Aves Chord… Carn… Least Concern #> 4 "Affen… Black… NA NA NA NA NA NA NA #> 5 "Afgha… Black… NA NA NA NA NA NA NA #> 6 "Afric… Grey,… 60 - 70… 3,600… Animal… Mamm… Chord… Herb… Threatened #> 7 "Afric… Black… 15 - 20… 1.4kg… Animal… Mamm… Chord… Omni… Least Concern #> 8 "Afric… Brown… 8 - 15 … 25g -… Animal… Amph… Chord… Carn… Least Concern #> 9 "Afric… Grey,… 60 - 70… 900kg… Animal… Mamm… Chord… Herb… Endangered #> 10 "Afric… Black… 15 - 20… 1.4kg… Animal… Mamm… Chord… Omni… Least Concern #> # ℹ 600 more rows #> # ℹ 39 more variables: order , scientific_name , skin_type , #> # habitat , predators , family , lifestyle , #> # average_litter_size , genus , top_speed , #> # favourite_food , main_prey , type , common_name , #> # group , size , distinctive_features , size_l , #> # origin , special_features , location , … The new datasets are: * [`data_animals()`](https://tidymodels.github.io/modeldatatoo/reference/data_animals.html) contains a long-form description of the animal (in the `text` column) as well as quite a bit of missing data and malformed fields. * [`data_chimiometrie_2019()`](https://tidymodels.github.io/modeldatatoo/reference/data_chimiometrie_2019.html) contains spectra measured at 550 (unknown) wavelengths, published as the challenge at the Chimiometrie 2019 conference. * [`data_elevators()`](https://tidymodels.github.io/modeldatatoo/reference/data_elevators.html) contains information on a subset of the elevators in New York City. Because those datasets are stored online, accessing them requires an active internet connection. We plan on using those datasets mostly for workshops and websites. The datasets in the modeldata package are part of the package directly, so they can be used everywhere (regardless of an active internet connection). We typically use them for package documentation. spatialsample[](#spatialsample) -------------------------------- spatialsample is a package for spatial resampling, extending the rsample framework to help create spatial extrapolation between your analysis and assessment data sets. The latest release of spatialsample includes nearest neighbor distance matching (NNDM) cross-validation via [`spatial_nndm_cv()`](https://spatialsample.tidymodels.org/reference/spatial_nndm_cv.html) . NNDM is a variant of leave-one-out cross-validation which assigns each observation to a single assessment fold, and then attempts to remove data from each analysis fold until the nearest neighbor distance distribution between assessment and analysis folds matches the nearest neighbor distance distribution between training data and the locations a model will be used to predict. [Proposed by Milà et al. (2022)](https://doi.org/10.1111/2041-210X.13851) , this method aims to provide accurate estimates of how well models will perform in the locations they will actually be predicting. This method was originally implemented in the CAST package and can now be used with spatialsample as well. Let’s use the Ames housing data and turn it from a regular tibble into a `sf` object for spatial data. library(spatialsample) data(ames, package = "modeldata") ames_sf <- sf::st_as_sf(ames, coords = c("Longitude", "Latitude"), crs = 4326) Let’s assume that we are building a model to predict observations similar to this subset of the data: ames_prediction_sites <- ames_sf[2001:2100, ] Let’s create NNDM cross-validation folds from a reduced training set as an example, just to keep things light. ames_folds <- spatial_nndm_cv(ames_sf[1:100, ], ames_prediction_sites) The resulting `rset` contains 100 splits of the data, always keeping 1 of the 100 data points in the assessment set. ames_folds #> # A tibble: 100 × 2 #> splits id #> #> 1 Fold001 #> 2 Fold002 #> 3 Fold003 #> 4 Fold004 #> 5 Fold005 #> 6 Fold006 #> 7 Fold007 #> 8 Fold008 #> 9 Fold009 #> 10 Fold010 #> # ℹ 90 more rows Starting with all other 99 points in the analysis set, points are excluded until the distribution of nearest neighbor distances from the analysis set to the assessment set matches that of nearest neighbor distances from the training set to the prediction sites. Looking at one of the splits, we can see the single assessment point, the points included in the analysis set, and the points excluded as the buffer. get_rsplit(ames_folds, 3) |> autoplot() ![](figs/unnamed-chunk-10-1.png) The `ames_fold` object can then be used with functions from the tune package as usual. tidymodels.org[](#tidymodelsorg) --------------------------------- The tidymodels website, [tidymodels.org](https://www.tidymodels.org/) , has been updated to use [Quarto](https://quarto.org/) . Things largely look the same as before but this change simplifies the build system which should make it easier for more people to contribute. This change to Quarto has also allowed us to improve the search functionality of the website. The tables for finding parsnip models, recipe steps, and broom tidiers at [https://www.tidymodels.org/find/](https://www.tidymodels.org/find/) now all list objects across all CRAN packages, not just tidymodels packages. This should make it much easier to find the right extension for your task, even if not implemented within tidymodels! And if it does not exist yet, open an issue on GitHub or browse the [developer documentation for extending tidymodels](https://www.tidymodels.org/learn/#category=developer%20tools) ! Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to extend our thanks to all of the contributors to tidymodels in the last quarter: * agua: [@gvelasq](https://github.com/gvelasq) . * broom: [@awcm0n](https://github.com/awcm0n) , [@gregmacfarlane](https://github.com/gregmacfarlane) , [@jwilliman](https://github.com/jwilliman) , [@mccarthy-m-g](https://github.com/mccarthy-m-g) , [@RoyalTS](https://github.com/RoyalTS) , [@simonpcouch](https://github.com/simonpcouch) , and [@ste-tuf](https://github.com/ste-tuf) . * desirability2: [@topepo](https://github.com/topepo) . * embed: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@naveranoc](https://github.com/naveranoc) . * probably: [@agormp](https://github.com/agormp) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@juliasilge](https://github.com/juliasilge) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * spatialsample: [@jamesgrecian](https://github.com/jamesgrecian) , [@mikemahoney218](https://github.com/mikemahoney218) , and [@nipnipj](https://github.com/nipnipj) . * tidymodels: [@forecastingEDs](https://github.com/forecastingEDs) , [@JosiahParry](https://github.com/JosiahParry) , and [@topepo](https://github.com/topepo) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # gmailr 2.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) gmailr 2.0.0 ============ ![](/blog/2023/06/gmailr-2-0-0/thumbnail-wd.jpg) Photo by [Hiroshi Kimura](https://unsplash.com/photos/VybzKEUMhbw)   2023/06/29   [gargle](/tags/gargle/) , [gmailr](/tags/gmailr/)   Jennifer Bryan We’re chuffed to announce the release of [gmailr](https://gmailr.r-lib.org/) 2.0.0. gmailr exposes the [Gmail API](https://developers.google.com/gmail/api/guides) from R. You can install it from CRAN with: install.packages("gmailr") The main goal of version 2.0.0 is to improve the ergonomics around auth. There is less need for fussy code around configuring an OAuth client and it’s easier to use gmailr in a non-interactive or deployed setting. There is also a major advance in the process of replacing legacy functions with versions that have a `gm_` prefix. The legacy functions still exist, but are now hard deprecated. Finally, gmailr no longer re-exports `%>%`, the magrittr pipe, now that we have `|>` in base R. You can see a full list of changes in the [release notes](https://github.com/r-lib/gmailr/releases/tag/v2.0.0) . library(gmailr) #> #> Attaching package: 'gmailr' #> The following object is masked from 'package:utils': #> #> history #> The following objects are masked from 'package:base': #> #> body, date, labels, message 😬 _Ouch! These name collisions are **exactly** why gmailr added a universal `gm_` prefix to all of its functions starting in v1.0.0. One day in the not-too-distant future we can remove the troublesome legacy functions._ OAuth client[](#oauth-client) ------------------------------ The Gmail API is more challenging to wrap, in terms of auth, than the APIs for Sheets, Drive, or BigQuery. That’s because the scopes (think: “permissions”) needed for the Gmail API are [regarded as extremely sensitive](https://developers.google.com/gmail/api/auth/scopes#scopes) , as well they should be. If a bad actor gains the ability to read and send email as you, that is considerably more damaging than them being able to modify your spreadsheets (which is also bad, to be sure, but considerably less bad). Email is particularly important, because most other services allow you to reset your password via email; if someone gets access to your email, they can quickly use that to access every other service you have a log in for. The heightened security around the Gmail API means that a wrapper package like gmailr can’t make auth “just work” as easily we can in other packages, such as googledrive. In particular, R users who want to use gmailr absolutely must provide their own _OAuth client_. In other packages, we can make this optional. gmailr v2.0.0 includes new features and documentation to reduce the pain around the OAuth client as much as possible: * [Set up an OAuth client](https://gmailr.r-lib.org/articles/oauth-client.html) is a new article with detailed instructions for creating and configuring an OAuth client. You might even say this provides an excruciating level of detail, but this process has proven to be tricky for many users. * There is now a default location for the JSON file that represents the OAuth client. It’s the location returned by `rappdirs::user_data_dir("gmailr")`. If you put the JSON file there, gmailr will find it automagically. rappdirs::user_data_dir("gmailr") |> list.files() #> [1] "client_secret_xxx-yyy.apps.googleusercontent.com.json" gm_default_oauth_client() #> [1] "/Users/jenny/Library/Application Support/gmailr/client_secret_xxx-yyy.apps.googleusercontent.com.json" [`gm_default_oauth_client()`](https://gmailr.r-lib.org/reference/gmailr-configuration.html) is the new function that implements this new feature as well as pre-existing support for providing this path via an environment variable. * If the OAuth client is configured for auto-discovery, it is no longer necessary to call [`gm_auth_configure()`](https://gmailr.r-lib.org/reference/gm_auth_configure.html) explicitly. That is taken care of internally, inside [`gm_auth()`](https://gmailr.r-lib.org/reference/gm_auth.html) . > library(gmailr) > > # 😲 OMG no more need to call gm_auth_configure() here! 🎉 > > gm_threads() The gmailr package is requesting access to your Google account. Enter '1' to start a new auth process or select a pre-authorized account. 1: Send me to the browser for a new auth process. 2: jenny@posit.co Selection: 2 * Conversely, if you happen to be providing an explicit user or service account token, `gm_auth(token =)` and `gm_auth(path =, subject =)`[1](#fn:1) no longer error if the OAuth client is not configured. Auth in a deployed or other non-interactive setting[](#auth-in-a-deployed-or-other-non-interactive-setting) ------------------------------------------------------------------------------------------------------------ The Gmail API is primarily intended for use on behalf of a regular Google user account. The gmailr package is designed to guide an interactive R user through a process in which they authenticate themselves to Google and authorize Gmail activities initiated from R. This is sometimes referred to as the “OAuth dance”.[2](#fn:2) But what about settings where there is no interactive user sitting around to do this dance, i.e. when gmailr-using code is deployed to a remote server or otherwise runs unattended? For most Google APIs, the standard advice is “use a service account”. But the Gmail API is special. To use a service account with the Gmail API basically requires that the service account has been delegated domain-wide authority. This is tricky for at least two reasons. First, this is only possible within a Google Workspace, i.e. it’s not available to personal Google accounts. Second, most Google Workspace admins will refuse to do this, for security reasons. Therefore, if you want to deploy a data product that uses gmailr, it’s extremely likely that you really do need to use a user token. This workflow has gotten dramatically easier in gmailr v2.0.0: * [Deploy a token](https://gmailr.r-lib.org/articles/deploy-a-token.html) is a new article describing how to capture a token interactively, then use it later, non-interactively. * [`gm_token_write()`](https://gmailr.r-lib.org/reference/gm_token_write.html) + [`gm_token_read()`](https://gmailr.r-lib.org/reference/gm_token_write.html) is a new matched pair of functions that facilitate writing an obfuscated token to disk then reloading that token in a deployed data product or in CI. * gmailr ships with [example code](https://github.com/r-lib/gmailr/tree/main/inst/deployed-token-demo) that uses this technique in a small Shiny app that sends email from a specific user account. See the contents of `system.file("deployed-token-demo", package = "gmailr")`. The heart of this approach is to first capture a token in an interactive session: gm_auth("user@example.com", cache = FALSE) # interactive OAuth dance in the browser happens HERE gm_token_write( path = ".secrets/gmailr-token.rds", key = "SUPER_SECRET_ENCRYPTION_KEY" ) then reload it in a subsequent non-interactive session: gm_auth(token = gm_token_read( ".secrets/gmailr-token.rds", key = "SUPER_SECRET_ENCRYPTION_KEY" )) Progress on The Great Renaming[](#progress-on-the-great-renaming) ------------------------------------------------------------------ Unfortunately, there is considerable overlap between some obvious function names in an email-related package (e.g. “body”, “date”, “message”) and pre-existing functions in base R (e.g.  [`body()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) , [`date()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) , [`message()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) ). From very early on, gmailr exported several functions with regrettable name collisions, as evidenced at the beginning of this post when we called [`library(gmailr)`](https://gmailr.r-lib.org) . In version 1.0.0 (released 2019-08-30), the process of addressing this problem kicked off. At that time, gmailr adopted a universal `gm_` prefix for its functions and soft deprecated the legacy functions. Here’s an indicative sample of the function replacements: * [`body()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) ➡️ [`gm_body()`](https://gmailr.r-lib.org/reference/gm_body.html) * [`date()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) ➡️ [`gm_date()`](https://gmailr.r-lib.org/reference/accessors.html) * [`message()`](https://gmailr.r-lib.org/reference/gmailr-deprecated.html) ➡️ [`gm_message()`](https://gmailr.r-lib.org/reference/gm_message.html) In version 2.0.0, the legacy functions are hard deprecated and you should expect them to be removed in the next release of gmailr. I don’t expect there to be much (any?) surviving usage of these functions, but it’s definitely time to eliminate any remaining usage. Use the native pipe[](#use-the-native-pipe) -------------------------------------------- gmailr is designed to be very pipe friendly and it leads to very natural code that builds up a message from its parts: msg <- gm_mime() |> gm_to("recipient@example.com") |> gm_from("sender@example.com") |> gm_subject("Hello, world!") |> gm_text_body("I come in peace.") gmailr predates the introduction of the native pipe, in R 4.1, and therefore, historically, it has re-exported `%>%`, the magrittr pipe, for user convenience. The magrittr pipe also featured heavily in gmailr’s documentation. In the v2.0.0 release, I’ve removed the magrittr dependency and now use the native pipe operator `|>` in all documentation (gmailr never used the pipe internally). The purrr package pioneered this maneuver, within the tidyverse, and gmailr uses the same techniques to resolve the tension between the new usage of the base pipe and the tidyverse policy of supporting older R versions. You can learn more about the pipe transition in the blog post [Differences between the base R and magrittr pipes](https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe/) . Acknowledgements[](#acknowledgements) -------------------------------------- A big thank you to all those who have contributed to gmailr since the v1.0.0 release: [@absuag](https://github.com/absuag) , [@aeburger](https://github.com/aeburger) , [@andresxmv](https://github.com/andresxmv) , [@batpigandme](https://github.com/batpigandme) , [@beib](https://github.com/beib) , [@careercoachme](https://github.com/careercoachme) , [@chuagh74](https://github.com/chuagh74) , [@cstangor](https://github.com/cstangor) , [@EeethB](https://github.com/EeethB) , [@enricodata](https://github.com/enricodata) , [@FJCC](https://github.com/FJCC) , [@hadley](https://github.com/hadley) , [@HadyShaaban](https://github.com/HadyShaaban) , [@ismayc](https://github.com/ismayc) , [@j450h1](https://github.com/j450h1) , [@janebunr](https://github.com/janebunr) , [@jcheng5](https://github.com/jcheng5) , [@JeffreyCHoover](https://github.com/JeffreyCHoover) , [@jennybc](https://github.com/jennybc) , [@jimhester](https://github.com/jimhester) , [@jonmichael-caldwell](https://github.com/jonmichael-caldwell) , [@jreid88](https://github.com/jreid88) , [@Karlheinzniebuhr](https://github.com/Karlheinzniebuhr) , [@kputschko](https://github.com/kputschko) , [@KryeKuzhinieri](https://github.com/KryeKuzhinieri) , [@laurenmarietta](https://github.com/laurenmarietta) , [@lwjohnst86](https://github.com/lwjohnst86) , [@maelle](https://github.com/maelle) , [@majazaloznik](https://github.com/majazaloznik) , [@maticabgd](https://github.com/maticabgd) , [@MCOtto](https://github.com/MCOtto) , [@meheszlev](https://github.com/meheszlev) , [@Mr-Hadoop-Hotshot](https://github.com/Mr-Hadoop-Hotshot) , [@Niekuba](https://github.com/Niekuba) , [@norcalbiostat](https://github.com/norcalbiostat) , [@Patrikios](https://github.com/Patrikios) , [@pschloss](https://github.com/pschloss) , [@pythiantech](https://github.com/pythiantech) , [@randy3k](https://github.com/randy3k) , [@ratnexa](https://github.com/ratnexa) , [@sanjmeh](https://github.com/sanjmeh) , [@sdisav](https://github.com/sdisav) , [@sommerhd-royals](https://github.com/sommerhd-royals) , [@statnmap](https://github.com/statnmap) , [@tariuk](https://github.com/tariuk) , [@tvroylandt](https://github.com/tvroylandt) , [@vinaybugz](https://github.com/vinaybugz) , and [@VincentGuyader](https://github.com/VincentGuyader) . * * * 1. The `subject` argument of [`gm_auth()`](https://gmailr.r-lib.org/reference/gm_auth.html) is also new and facilitates the use of a service account to impersonate a user. [↩︎](#fnref:1) 2. The full OAuth dance is not necessary in subsequent R sessions, though, by default gmailr is very conservative and asks for permission to use and refresh an existing token. This is, of course, configurable. [↩︎](#fnref:2) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Package spring cleaning [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Package spring cleaning ======================= ![](/blog/2023/06/spring-cleaning-2023/thumbnail-wd.jpg) Photo by [Anthony Delanoix](https://unsplash.com/photos/urUdKCxsTUI)   2023/06/07   [package maintenance](/tags/package-maintenance/) , [spring cleaning](/tags/spring-cleaning/)   Andy Teucher When Spring arrives in the Northern hemisphere, the sun’s rays reach into the dark corners and illuminate the dust that has been gathering over the winter. This is when our thoughts start to turn to Spring cleaning — a time to clear out the clutter that has accumulated over the past year. It represents a fresh start and a new beginning, and leaves us feeling rejuvenated and ready to take on the rest of the year. This applies not only to our homes, but also to the code that we maintain — there are often bits and pieces that we know need attention but never seem to make it to the top of the priority list. Doing this kind of work isn’t necessarily only about adopting good practices or increasing the quality of your code — it can also be about adding value through standardization. Most developers only work sporadically on a particular package. For some it’s because they work on a lot of packages, while for many it’s because package development is not their main job. When you return to a package after a long gap, there is potential for a lot of friction (and dread/procrastination) as you get re-oriented to its idiosyncrasies. Making the occasional pass through your packages and looking for opportunities to adopt current, shared practices can make it easier to dip in and out of different packages. The tidyverse team at Posit has a practice of tackling Spring Cleaning together - we set aside a week every year to work in a semi-structured way to efficiently take care of a common list of package maintenance tasks. We find that setting a time for them and doing them all together during one week is an effective, and more fun, way to get them done. We recently completed our 2023 Spring Cleaning and thought it might be fun to share our process. I’ll also show off a new feature we’ve built in to the [latest version of usethis](https://usethis.r-lib.org/news/index.html#usethis-220) that will help you organize your own Spring Cleaning. Feel free to [jump straight there](#spring-cleaning-and-you) if you want to skip the back story (don’t you wish recipe blogs had this feature?). Preparation[](#preparation) ---------------------------- Early in the new year, we set aside the time in our calendars for Spring Cleaning — this way everyone knows that it’s coming up and can make sure they have cleared the space in their schedules (and their minds) to focus on it. We prepare for the week by creating a list of things we want to take care of in our packages. Rather than adding features or fixing bugs, these tasks are usually about bringing things up to current standards or best practices, and include things like updating tests to the latest testthat version, updating pkgdown templates, and adding alt-text to images in pkgdown sites. Not surprisingly, this year a lot of the upkeep was related to our [recent rebrand from RStudio to Posit](https://posit.co/blog/rstudio-is-now-posit/) — things like updating the copyright holder and author email addresses, and using updated logos without the old rstudio.com website on them. We start off the week with a kickoff meeting on Monday morning. We go through the checklist with everybody and refine what’s in it, making sure everybody has had input. Because we maintain so many packages, we have a spreadsheet where we keep track of the packages that are undergoing spring cleaning, and people can assign themselves to packages and mark them as completed when they’re done. Checklists, checklists, checklists[](#checklists-checklists-checklists) ------------------------------------------------------------------------ We formalize these tasks into a checklist ( [who doesn’t love checklists](https://atulgawande.com/book/the-checklist-manifesto/) ) via a tidyverse-focused function in usethis called `use_tidy_upkeep_issue()`. If you’re a package developer and you use [`use_release_issue()`](https://usethis.r-lib.org/reference/use_release_issue.html) , this will look familiar: it opens an issue in the package’s GitHub repository with a checklist of tasks to guide us through what needs to be done to bring it up to current tidyverse standards. We update the function with the current year’s checklist just prior to starting (and sometimes during) Spring Cleaning. Package maintainers then install the development version of usethis to get the current checklist, and call `usethis::use_tidy_upkeep_issue()` in their package to create the issue. If there are any tasks that aren’t relevant to that particular repo it’s easy to just edit the issue and remove it. To be really meta, here is the 2023 Spring Cleaning [upkeep issue for usethis](https://github.com/r-lib/usethis/issues/1791) , created by usethis: [![2023 Upkeep Issue for usethis](img/usethis-upkeep-issue.png)](https://github.com/r-lib/usethis/issues/1791) 2023 Upkeep Issue for usethis We separated the tasks into “Necessary” and “Optional”. The necessary tasks were those we needed to complete for all of our packages, and also were simple enough that we could be sure we would able to complete them. The optional items were those that were nice to have, and/or would take longer to complete. We try to complete the work, including reviewing and merging any related [pull requests](https://github.com/tidymodels/dials/pull/275) , all within the week, with the intention of closing the upkeep issue by Friday. Wrapup[](#wrapup) ------------------ Finally, we end the week with a wrap-up meeting - we do a retrospective on what worked, what didn’t, and what we would change for next time. For example, we found that a couple of items on this year’s checklist that were too complex to complete within the week, especially across many repos. So we decided to start a practice of converting those “too big” tasks into issues of their own — you can see an example in the [testthat upkeep issue](https://github.com/r-lib/testthat/issues/1749) . This makes it more likely that we can cleanly complete the checklist but still flag those lingering things we would like to finish. We also try to have a little fun during the wrap-up meeting! I made a small R package called [chatrbox](https://github.com/ateucher/chatrbox) that uses [ChatGPT](https://openai.com/blog/chatgpt) to generate R-themed Spring Cleaning text snippets. And Tracy Teal used [quarto](https://quarto.org/) to make certificates of achievement for each of us, complete with inspirational messages made with chatrbox! ![A certificate of excellence in Spring Cleaning for George Stagg, with AI-generated text in the form of a tweet about software licensing in the style of Shakespeare. The generated text says: "Of software fair, be wary and take heed, For licensing terms doth often mislead. Choose wisely, lest thou shouldst freely bruise." #SoftwareLicensing #ShakespeareanTweets](img/george-certificate.png) Spring cleaning and you![](#spring-cleaning-and-you) ----------------------------------------------------- In [version 2.2.0 of usethis](https://usethis.r-lib.org/news/index.html#usethis-220) , we have created a general purpose [`use_upkeep_issue()`](https://usethis.r-lib.org/reference/use_upkeep_issue.html) function for package authors to use if they wish to do a Spring Cleaning of their own. It is a fairly opinionated list of tasks but we believe taking care of them will generally make your package better, easier to maintain, and more enjoyable for your users. Some of the tasks are meant to be performed only once (and once completed shouldn’t show up in subsequent lists), and some should be reviewed periodically. If you want to include additional tasks, you can add an (unexported) function named `upkeep_bullets()` to your own package that returns a character vector of tasks. These will be added to your upkeep checklist. Here is an example of an upkeep issue I created for my package rmapshaper. I created an internal function `upkeep_bullets()` in the package, with an extra bullet I wanted to add to the upkeep issue: upkeep_bullets <- function() "Update bundled mapshaper node library." And then called `use_upkeep_issue()` in my rmapshaper package directory: devtools::load_all() #> ℹ Loading rmapshaper usethis::use_upkeep_issue() #> ✔ Setting active project to '/Users/andyteucher/dev/ateucher/rmapshaper' #> • Open URL 'https://github.com/ateucher/rmapshaper/issues/160' [![Upkeep issue for rmapshaper](img/rmapshaper-upkeep-issue.png)](https://github.com/ateucher/rmapshaper/issues/160) Upkeep issue for rmapshaper In a fun confluence of events, while working on this post I attended an [rOpenSci coworking session](https://ropensci.org/events/coworking-2023-05/) where the topic of the day was spring cleaning! We chatted about the benefits of regular upkeep, and what types of tasks make good spring cleaning issues. It was really inspiring and validating to connect with other people tackling maintenance like this. We hope that this will provide a starting point, and motivate you to take care of those nagging maintenance issues, whether it be in the Spring (whenever that is in your part of the world), or any other time of the year. We’d love to hear if you find this helpful, or if there’s a way that it could be better, please [let us know](https://github.com/r-lib/usethis/issues) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # purrr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Purrr [![](/blog/2023/05/purrr-walk-this-way/thumbnail-sq.jpg)](/blog/2023/05/purrr-walk-this-way/) [`purrr::walk()` this way](/blog/2023/05/purrr-walk-this-way/) [learn](/categories/learn) Mara Averick How to use `purrr::walk()` to write many files, featuring file-system navigation with the fs package. [Read more ...](/blog/2023/05/purrr-walk-this-way/) 2023/05/26 [![](/blog/2022/12/purrr-1-0-0/thumbnail-sq.jpg)](/blog/2022/12/purrr-1-0-0/) [purrr 1.0.0](/blog/2022/12/purrr-1-0-0/) [package](/categories/package) Hadley Wickham purrr 1.0.0 brings a basket of updates. We deprecated a number of seldom used functions to hone in on the core purpose of purrr and implemented a swath of new features including progress bars, improved error reporting, and much much more! [Read more ...](/blog/2022/12/purrr-1-0-0/) 2022/12/20 [![](/blog/2019/02/purrr-0-3-0/purrr-0-3-0-sq.jpg)](/blog/2019/02/purrr-0-3-0/) [purrr 0.3.0](/blog/2019/02/purrr-0-3-0/) [package](/categories/package) Lionel Henry purrr 0.3.0 is now on CRAN. [Read more ...](/blog/2019/02/purrr-0-3-0/) 2019/02/06 [![](/blog/2017/08/purrr-0.2.3/purrr-0.2.3-sq.jpg)](/blog/2017/08/purrr-0.2.3/) [purrr 0.2.3](/blog/2017/08/purrr-0.2.3/) [package](/categories/package) Lionel Henry A new version of purrr is on CRAN! It features a new family of generic mappers, a tool for tidy plucking of deep data structures, and many other features and fixes. [Read more ...](/blog/2017/08/purrr-0.2.3/) 2017/08/15 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # desirability [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Desirability [![](/blog/2023/05/desirability2/thumbnail-sq.jpg)](/blog/2023/05/desirability2/) [desirability2](/blog/2023/05/desirability2/) [package](/categories/package) Max Kuhn The desirability2 package, for multivariable optimization, is now on CRAN. [Read more ...](/blog/2023/05/desirability2/) 2023/05/17 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # desirability2 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) desirability2 ============= ![](/blog/2023/05/desirability2/thumbnail-wd.jpg) Photo by [Joel Naren](https://unsplash.com/photos/8cvksz5mmnE)   2023/05/17   [tidymodels](/tags/tidymodels/) , [desirability](/tags/desirability/) , [optimization](/tags/optimization/)   Max Kuhn We’re tickled pink to announce the release of [desirability2](http://desirability2.tidymodels.org) (version 0.0.1). You can install it from CRAN with: install.packages("desirability2") This blog post will introduce you to the package and desirability functions. Let’s load some packages! library(desirability2) library(dplyr) library(ggplot2) [Desirability functions](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C7&q=%22desirability+functions%22) are tools that can be used to rank or optimize multiple characteristics at once. They are intuitive and easy to use. There are a few R packages that implement them, including [desirability](http://cran.r-project.org/package=desirability) and [desiR](http://cran.r-project.org/package=desiR) . We have a new one, [desirability2](http://cran.r-project.org/package=desirability2) , with an interface conducive to being used in-line via dplyr pipelines. Let’s demonstrate that by looking at an application. Suppose we created a classification model and produced multiple metrics on how well it classifies new data. We measured the area under the ROC curve and the binomial log-loss statistic in this example. There are about 300 different model configurations that we investigated via tuning. The results from the tuning process were: classification_results ## # A tibble: 298 × 5 ## mixture penalty mn_log_loss roc_auc num_features ## ## 1 0 0.1 0.199 0.869 211 ## 2 0 0.0788 0.196 0.870 211 ## 3 0 0.0621 0.194 0.871 211 ## 4 0 0.0489 0.192 0.872 211 ## 5 0 0.0386 0.191 0.873 211 ## 6 0 0.0304 0.190 0.873 211 ## 7 0 0.0240 0.188 0.874 211 ## 8 0 0.0189 0.188 0.874 211 ## 9 0 0.0149 0.187 0.874 211 ## 10 0 0.0117 0.186 0.874 211 ## # ℹ 288 more rows If we were interested in the best area under the ROC curve: classification_results |> slice_max(roc_auc, n = 1) ## # A tibble: 1 × 5 ## mixture penalty mn_log_loss roc_auc num_features ## ## 1 0.222 0.00574 0.185 0.876 86 However, there are different optimal settings when the log-likelihood is considered: classification_results |> slice_min(mn_log_loss, n = 1) ## # A tibble: 1 × 5 ## mixture penalty mn_log_loss roc_auc num_features ## ## 1 1 0.000853 0.184 0.876 103 Are the two metrics related? Here’s a plot of the data: classification_results |> ggplot(aes(roc_auc, mn_log_loss, col = num_features)) + geom_point(alpha = 1/2) ![plot of chunk unnamed-chunk-7](figure/unnamed-chunk-7-1.svg) We colored the point using the number of features used in the model. Fewer predictors are better; we’d like to factor that into the tuning parameter selection. To optimize them all at once, desirability functions map their values to be between zero and one (with the latter being the most desirable). For the ROC scores, a value of 1.0 is best, and we may not consider a model with an AUC of less than 0.80. We can use desirability2’s [`d_max()`](http://desirability2.tidymodels.org/reference/inline_desirability.html) function to translate these values to desirability: classification_results %>% mutate(roc_d = d_max(roc_auc, high = 1, low = 0.8)) %>% ggplot(aes(roc_auc, roc_d)) + geom_line() + geom_point() + lims(y = 0:1) ![plot of chunk unnamed-chunk-8](figure/unnamed-chunk-8-1.svg) Note that all model configurations with ROC AUC scores below 0.80 have zero desirability. Since we want to reduce loss, we can use `d_min()` to show a curve where smaller is better. For this specification, we’ll use the min and max values as defined by the data, by setting `use_data = TRUE`: classification_results %>% mutate( roc_d = d_max(roc_auc, high = 1, low = 0.8), loss_d = d_min(mn_log_loss, use_data = TRUE) ) %>% ggplot(aes(mn_log_loss, loss_d)) + geom_line() + geom_point() + lims(y = 0:1) ![plot of chunk unnamed-chunk-9](figure/unnamed-chunk-9-1.svg) Finally, we can factor in the number of features. Arguably this is more important to use than the other two outcomes; we will make this curve nonlinear so that it becomes more challenging to be desirable as the number of features increases. For this, we’ll use the `scale` option to `d_min()`, where larger values make the criteria more difficult to satisfy: classification_results %>% mutate( roc_d = d_max(roc_auc, high = 1, low = 0.8), loss_d = d_min(mn_log_loss, use_data = TRUE), feat_d = d_min(num_features, low = 0, high = 100, scale = 2) ) %>% ggplot(aes(num_features, feat_d)) + geom_line() + geom_point() + lims(y = 0:1) ![plot of chunk unnamed-chunk-10](figure/unnamed-chunk-10-1.svg) Combining these components into a single criterion using the geometric mean is common. Using this statistic has the side effect that any criteria with zero desirability make the overall desirability zero (since the geometric mean multiples the values). There is a function called [`d_overall()`](http://desirability2.tidymodels.org/reference/d_overall.html) that can be used with dplyr’s `across()` function. Sorting by overall desirability gives us tuning parameter values (`mixture` and `penalty`) that are best for this combination of criteria. classification_results %>% mutate( roc_d = d_max(roc_auc, high = 1, low = 0.8), loss_d = d_min(mn_log_loss, use_data = TRUE), feat_d = d_min(num_features, low = 0, high = 100, scale = 2), overall = d_overall(across(ends_with("_d"))) ) %>% slice_max(overall, n = 5) ## # A tibble: 5 × 9 ## mixture penalty mn_log_loss roc_auc num_features roc_d loss_d feat_d overall ## ## 1 1 0.00924 0.200 0.859 15 0.295 0.815 0.722 0.558 ## 2 0.667 0.0117 0.199 0.862 18 0.311 0.827 0.672 0.557 ## 3 0.667 0.0149 0.201 0.858 14 0.291 0.802 0.740 0.557 ## 4 0.889 0.00924 0.199 0.861 18 0.305 0.825 0.672 0.553 ## 5 0.889 0.0117 0.201 0.857 14 0.285 0.801 0.740 0.553 That’s it! That’s the package. The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # optimization [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Optimization [![](/blog/2023/05/desirability2/thumbnail-sq.jpg)](/blog/2023/05/desirability2/) [desirability2](/blog/2023/05/desirability2/) [package](/categories/package) Max Kuhn The desirability2 package, for multivariable optimization, is now on CRAN. [Read more ...](/blog/2023/05/desirability2/) 2023/05/17 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # magrittr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Magrittr [![](/blog/2023/04/base-vs-magrittr-pipe/thumbnail-sq.jpg)](/blog/2023/04/base-vs-magrittr-pipe/) [Differences between the base R and magrittr pipes](/blog/2023/04/base-vs-magrittr-pipe/) [other](/categories/other) Hadley Wickham A discussion of the (relatively minor) differences between the native R pipe, `|>`, and the magrittr pipe, `%>%`. [Read more ...](/blog/2023/04/base-vs-magrittr-pipe/) 2023/04/21 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Teaching the tidyverse in 2023 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Teaching the tidyverse in 2023 ============================== ![](/blog/2023/08/teach-tidyverse-23/thumbnail-wd.jpg) Photo by [Scott Evans](https://unsplash.com/photos/ScoYEG5LEgc)   2023/08/07   [teaching](/tags/teaching/) , [tidyverse](/tags/tidyverse/)   Mine Çetinkaya-Rundel Another year, another roundup of tidyverse updates, through the lens of an educator. As with previous [teaching the tidyverse posts](/blog/2021/08/teach-tidyverse-2021/) , much of what is discussed in this blog post has already been covered in package update posts, however the goal of this roundup is to summarize the highlights that are most relevant to teaching data science with the tidyverse, particularly to new learners. Specifically, I’ll discuss: * [Resource refresh](#resource-refresh) * [Nine core packages in tidyverse 2.0.0](#nine-core-packages-in-tidyverse-200) * [Conflict resolution in the tidyverse](#conflict-resolution-in-the-tidyverse) * [Improved and expanded `*_join()` functionality](#improved-and-expanded-_join-functionality) * [Per operation grouping](#per-operation-grouping) * [Quality of life improvements to `case_when()` and `if_else()`](#quality-of-life-improvements-to-case_when-and-if_else) * [New syntax for separating columns](#new-syntax-for-separating-columns) * [New argument for line geoms: linewidth](#new-argument-for-line-geoms-linewidth) * [Other highlights](#other-highlights) * [Coming up](#coming-up) And different from previous posts on this topic, this one comes with a video! If you’d like a live demo of the code examples, and a few more additional tips along the way, you can watch the video below. Throughout this blog post you’ll encounter some code chunks with the comment `previously`, indicating what you used to do in the tidyverse. Often these will be coupled with chunks with the comment `now, optionally`, indicating what you _can_ now do with the tidyverse. And rarely, they will be coupled with chunks with the comment `now`, indicating what you _should_ do instead now with the tidyverse. Let’s get started with the obligatory… library(tidyverse) #> ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ── #> ✔ dplyr 1.1.2 ✔ readr 2.1.4 #> ✔ forcats 1.0.0 ✔ stringr 1.5.0 #> ✔ ggplot2 3.4.2 ✔ tibble 3.2.1 #> ✔ lubridate 1.9.2 ✔ tidyr 1.3.0 #> ✔ purrr 1.0.1 #> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ℹ Use the conflicted package () to force all conflicts to become errors And, let’s also load the [palmerpenguins](https://allisonhorst.github.io/palmerpenguins/) package that we will use in examples. library(palmerpenguins) Resource refresh[](#resource-refresh) -------------------------------------- R for Data Science, 2nd Edition is out! [This blog post](/blog/2023/07/r4ds-2e/) (and the [book’s preface](https://r4ds.hadley.nz/preface-2e.html) ) outlines updates since the first edition. Updates to the book served as the motivation for many of the changes mentioned in the remainder of this post as as well as on the Tidyverse blog over the last year. Now that the book is out, you can expect the pace of change to slow down again for a while, which means plenty of time for phasing these changes into your teaching materials. One change in the 2nd Edition that will most likely affect almost all of your teaching materials is the use of the native R pipe (`|>`) instead of the magrittr pipe (`%>%`). If you’re not familiar with the similarities and differences between these operators, I recommend reading [this comparison blog post](https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe/) . And I strongly recommend making this update since it will allow students to perform piped operations with any R function, and hence allow them to keep their data pipeline workflows regardless of whether the next package they learn is from the tidyverse (or package that uses tidyverse principles) or not. Nine core packages in tidyverse 2.0.0[](#nine-core-packages-in-tidyverse-200) ------------------------------------------------------------------------------ The main update in tidyverse 2.0.0, which was released in March 2023, is that it [lubridate](https://lubridate.tidyverse.org/) is now a core tidyverse package. The lubridate package that makes it easier to do the things R does with date-times, is now a core tidyverse package. So, while many of your scripts in the past may have started with # previously library(tidyverse) library(lubridate) you can now just do # now library(tidyverse) and the lubridate package will be loaded as well. If you, like me, use a graphic like the one below that maps the core tidyverse packages to phases of the data science cycle, here is an updated graphic including lubridate. ![](images/data-science.png) Conflict resolution in the tidyverse[](#conflict-resolution-in-the-tidyverse) ------------------------------------------------------------------------------ You may have also noticed that the package loading message for the tidyverse has been updated as well, and now advertises the [conflicted](https://conflicted.r-lib.org/) package. #> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ℹ Use the conflicted package () to force all conflicts to become errors Conflict resolution in R, i.e., what to do if multiple packages that are loaded in a session have functions with the same name, can get tricky, and the conflicted package is designed to help with that. R’s default conflict resolution gives precedence to the most recently loaded package. For example, if you use the filter function before loading the tidyverse, R will use [`stats::filter()`](https://rdrr.io/r/stats/filter.html) : penguins |> filter(species == "Adelie") #> Error in eval(expr, envir, enclos): object 'species' not found However, after loading the tidyverse, when you call [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) , R will _silently_ choose [`dplyr::filter()`](https://dplyr.tidyverse.org/reference/filter.html) : penguins |> filter(species == "Adelie") #> # A tibble: 152 × 8 #> species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g #> #> 1 Adelie Torgersen 39.1 18.7 181 3750 #> 2 Adelie Torgersen 39.5 17.4 186 3800 #> 3 Adelie Torgersen 40.3 18 195 3250 #> 4 Adelie Torgersen NA NA NA NA #> 5 Adelie Torgersen 36.7 19.3 193 3450 #> 6 Adelie Torgersen 39.3 20.6 190 3650 #> 7 Adelie Torgersen 38.9 17.8 181 3625 #> 8 Adelie Torgersen 39.2 19.6 195 4675 #> 9 Adelie Torgersen 34.1 18.1 193 3475 #> 10 Adelie Torgersen 42 20.2 190 4250 #> # ℹ 142 more rows #> # ℹ 2 more variables: sex , year This silent conflict resolution approach works fine until it doesn’t, and then it can be very frustrating to debug. The conflicted package does not allow for silent conflict resolution: library(conflicted) penguins |> filter(species == "Adelie") #> Error: #> ! [conflicted] filter found in 2 packages. #> Either pick the one you want with `::`: #> • dplyr::filter #> • stats::filter #> Or declare a preference with `conflicts_prefer()`: #> • `conflicts_prefer(dplyr::filter)` #> • `conflicts_prefer(stats::filter)` You can, of course, use [`dplyr::filter()`](https://dplyr.tidyverse.org/reference/filter.html) but if you have a bunch of data wrangling pipelines, which is likely the case if you’re teaching data wrangling, it can get pretty busy. Instead, with conflicted, you can explicitly declare which [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) you want to use at the beginning (of a session, of a script, or of an R Markdown or Quarto file) with [`conflicts_prefer()`](https://conflicted.r-lib.org/reference/conflicts_prefer.html) : conflicts_prefer(dplyr::filter) #> [conflicted] Will prefer dplyr::filter over any other package. penguins |> filter(species == "Adelie") #> # A tibble: 152 × 8 #> species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g #> #> 1 Adelie Torgersen 39.1 18.7 181 3750 #> 2 Adelie Torgersen 39.5 17.4 186 3800 #> 3 Adelie Torgersen 40.3 18 195 3250 #> 4 Adelie Torgersen NA NA NA NA #> 5 Adelie Torgersen 36.7 19.3 193 3450 #> 6 Adelie Torgersen 39.3 20.6 190 3650 #> 7 Adelie Torgersen 38.9 17.8 181 3625 #> 8 Adelie Torgersen 39.2 19.6 195 4675 #> 9 Adelie Torgersen 34.1 18.1 193 3475 #> 10 Adelie Torgersen 42 20.2 190 4250 #> # ℹ 142 more rows #> # ℹ 2 more variables: sex , year Getting back to the package loading message… It can be tempting, particularly in a teaching scenario, particularly to an audience of new learners, and particularly if you teach with slides and messages take up valuable slide real estate, I would urge you to not hide startup messages from teaching materials. Instead, address them early on to: 1. Encourage reading and understanding messages, warnings, and errors – teaching people to read error messages is hard enough, it’s going to be even harder if you’re not modeling that to them. 2. Help during hard-to-debug situations resulting from base R’s silent conflict resolution – because, let’s face it, someone in your class, if not you during a live-coding session, will see that pesky object not found error at some point when using [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) . Improved and expanded `*_join()` functionality[](#improved-and-expanded-_join-functionality) --------------------------------------------------------------------------------------------- The [dplyr](https://dplyr.tidyverse.org/) package has long had the [`*_join()` family of functions](https://dplyr.tidyverse.org/articles/two-table.html) for joining data frames. dplyr 1.1.0 introduced a [bunch of extensions](https://www.tidyverse.org/blog/2023/01/dplyr-1-1-0-joins/) that bring joins closer to the power available in other systems like SQL and `data.table`. ### `join_by()`[](#join_by) New functionality for join functions includes a new [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) function for the `by` argument. So, while in the past your code may have looked like the following: # previously *_join( x, y, by = c("" = "") ) you can now do: # now, optionally *_join( x, y, by = join_by( == ) ) For example, suppose you have the following information on the three islands we have penguins from: islands <- tribble( ~name, ~coordinates, "Torgersen", "64°46′S 64°5′W", "Biscoe", "65°26′S 65°30′W", "Dream", "64°44′S 64°14′W" ) islands #> # A tibble: 3 × 2 #> name coordinates #> #> 1 Torgersen 64°46′S 64°5′W #> 2 Biscoe 65°26′S 65°30′W #> 3 Dream 64°44′S 64°14′W You can join this to the penguins data frame by matching the `island` column in the penguins data frame to the `name` column in the islands data frame: penguins |> left_join( islands, by = join_by(island == name) ) |> select(species, island, coordinates) #> # A tibble: 344 × 3 #> species island coordinates #> #> 1 Adelie Torgersen 64°46′S 64°5′W #> 2 Adelie Torgersen 64°46′S 64°5′W #> 3 Adelie Torgersen 64°46′S 64°5′W #> 4 Adelie Torgersen 64°46′S 64°5′W #> 5 Adelie Torgersen 64°46′S 64°5′W #> 6 Adelie Torgersen 64°46′S 64°5′W #> 7 Adelie Torgersen 64°46′S 64°5′W #> 8 Adelie Torgersen 64°46′S 64°5′W #> 9 Adelie Torgersen 64°46′S 64°5′W #> 10 Adelie Torgersen 64°46′S 64°5′W #> # ℹ 334 more rows While `by = c("island" = "name")` would still work, I would recommend teaching [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) over `by` so that: 1. You can read it out loud as “where x is equal to y”, just like in other logical statements where `==` is pronounced as “is equal to”. 2. You don’t have to worry about `by = c(x = y)` (which is invalid) vs. `by = c(x = "y")` (which is valid) vs. `by = c("x" = "y")` (which is also valid). In fact, for succinctness, you might avoid the argument name and express this as: penguins |> left_join(islands, join_by(island == name)) ### Handling various matches[](#handling-various-matches) The `*_join()` functions now have additional arguments for handling `multiple` matches and `unmatched` rows as well as for specifying the `relationship` between the two data frames. So, while in the past your code may have looked like the following: # previously *_join( x, y, by ) you can now do: # now, optionally *_join( x, y, by, multiple = "all", unmatched = "drop", relationship = NULL ) Let’s set up three data frames to demonstrate the new functionality: * Information about three penguins, one row per `samp_id`: three_penguins <- tribble( ~samp_id, ~species, ~island, 1, "Adelie", "Torgersen", 2, "Gentoo", "Biscoe", 3, "Chinstrap", "Dream" ) three_penguins #> # A tibble: 3 × 3 #> samp_id species island #> #> 1 1 Adelie Torgersen #> 2 2 Gentoo Biscoe #> 3 3 Chinstrap Dream * Information about weight measurements of these penguins, one row per `samp_id`, `meas_id` combination: weight_measurements <- tribble( ~samp_id, ~meas_id, ~body_mass_g, 1, 1, 3220, 1, 2, 3250, 2, 1, 4730, 2, 2, 4725, 3, 1, 4000, 3, 2, 4050 ) weight_measurements #> # A tibble: 6 × 3 #> samp_id meas_id body_mass_g #> #> 1 1 1 3220 #> 2 1 2 3250 #> 3 2 1 4730 #> 4 2 2 4725 #> 5 3 1 4000 #> 6 3 2 4050 * Information about flipper measurements of these penguins, one row per `samp_id`, `meas_id` combination: flipper_measurements <- tribble( ~samp_id, ~meas_id, ~flipper_length_mm, 1, 1, 193, 1, 2, 195, 2, 1, 214, 2, 2, 216, 3, 1, 203, 3, 2, 203 ) flipper_measurements #> # A tibble: 6 × 3 #> samp_id meas_id flipper_length_mm #> #> 1 1 1 193 #> 2 1 2 195 #> 3 2 1 214 #> 4 2 2 216 #> 5 3 1 203 #> 6 3 2 203 One-to-many relationships don’t require extra care, they just work: three_penguins |> left_join(weight_measurements, join_by(samp_id)) #> # A tibble: 6 × 5 #> samp_id species island meas_id body_mass_g #> #> 1 1 Adelie Torgersen 1 3220 #> 2 1 Adelie Torgersen 2 3250 #> 3 2 Gentoo Biscoe 1 4730 #> 4 2 Gentoo Biscoe 2 4725 #> 5 3 Chinstrap Dream 1 4000 #> 6 3 Chinstrap Dream 2 4050 However, many-to-many relationships require some extra care. For example, if we join the `three_penguins` data frame to the `flipper_measurements` data frame, we get a warning: weight_measurements |> left_join(flipper_measurements, join_by(samp_id)) #> Warning in left_join(weight_measurements, flipper_measurements, join_by(samp_id)): Detected an unexpected many-to-many relationship between `x` and `y`. #> ℹ Row 1 of `x` matches multiple rows in `y`. #> ℹ Row 1 of `y` matches multiple rows in `x`. #> ℹ If a many-to-many relationship is expected, set `relationship = #> "many-to-many"` to silence this warning. #> # A tibble: 12 × 5 #> samp_id meas_id.x body_mass_g meas_id.y flipper_length_mm #> #> 1 1 1 3220 1 193 #> 2 1 1 3220 2 195 #> 3 1 2 3250 1 193 #> 4 1 2 3250 2 195 #> 5 2 1 4730 1 214 #> 6 2 1 4730 2 216 #> 7 2 2 4725 1 214 #> 8 2 2 4725 2 216 #> 9 3 1 4000 1 203 #> 10 3 1 4000 2 203 #> 11 3 2 4050 1 203 #> 12 3 2 4050 2 203 We get a warning about unexpected many-to-many relationships (unexpected because we didn’t specify this type of relationship in our join call), and the warning suggests setting `relationship = "many-to-many"`. And note that we went from 6 rows (measurements) to 12, which is also unexpected. weight_measurements |> left_join(flipper_measurements, join_by(samp_id), relationship = "many-to-many") #> # A tibble: 12 × 5 #> samp_id meas_id.x body_mass_g meas_id.y flipper_length_mm #> #> 1 1 1 3220 1 193 #> 2 1 1 3220 2 195 #> 3 1 2 3250 1 193 #> 4 1 2 3250 2 195 #> 5 2 1 4730 1 214 #> 6 2 1 4730 2 216 #> 7 2 2 4725 1 214 #> 8 2 2 4725 2 216 #> 9 3 1 4000 1 203 #> 10 3 1 4000 2 203 #> 11 3 2 4050 1 203 #> 12 3 2 4050 2 203 With `relationship = "many-to-many"`, we no longer get a warning. However, the “explosion of rows” issue is still there. Addressing that requires rethinking what we join the two data frames by: weight_measurements |> left_join(flipper_measurements, join_by(samp_id, meas_id)) #> # A tibble: 6 × 4 #> samp_id meas_id body_mass_g flipper_length_mm #> #> 1 1 1 3220 193 #> 2 1 2 3250 195 #> 3 2 1 4730 214 #> 4 2 2 4725 216 #> 5 3 1 4000 203 #> 6 3 2 4050 203 We can see that while the warning nudged us towards setting `relationship = "many-to-many"`, turns out the correct way to address the problem was to join by both `samp_id` and `meas_id`. We’ll wrap up our discussion on new functionality for handling `unmatched` cases. We’ll create one more data frame (`four_penguins`) to exemplify this: four_penguins <- tribble( ~samp_id, ~species, ~island, 1, "Adelie", "Torgersen", 2, "Gentoo", "Biscoe", 3, "Chinstrap", "Dream", 4, "Adelie", "Biscoe" ) four_penguins #> # A tibble: 4 × 3 #> samp_id species island #> #> 1 1 Adelie Torgersen #> 2 2 Gentoo Biscoe #> 3 3 Chinstrap Dream #> 4 4 Adelie Biscoe If we just join `weight_measurements` to `four_penguins`, the unmatched fourth penguin silently disappears, which is less than ideal, particularly in a more realistic scenario with many more observations: weight_measurements |> left_join(four_penguins, join_by(samp_id)) #> # A tibble: 6 × 5 #> samp_id meas_id body_mass_g species island #> #> 1 1 1 3220 Adelie Torgersen #> 2 1 2 3250 Adelie Torgersen #> 3 2 1 4730 Gentoo Biscoe #> 4 2 2 4725 Gentoo Biscoe #> 5 3 1 4000 Chinstrap Dream #> 6 3 2 4050 Chinstrap Dream Setting `unmatched = "error"` protects you from accidentally dropping rows: weight_measurements |> left_join(four_penguins, join_by(samp_id), unmatched = "error") #> Error in `left_join()`: #> ! Each row of `y` must be matched by `x`. #> ℹ Row 4 of `y` was not matched. Once you see the error message, you can decide how to handle the unmatched rows, e.g., explicitly drop them. weight_measurements |> left_join(four_penguins, join_by(samp_id), unmatched = "drop") #> # A tibble: 6 × 5 #> samp_id meas_id body_mass_g species island #> #> 1 1 1 3220 Adelie Torgersen #> 2 1 2 3250 Adelie Torgersen #> 3 2 1 4730 Gentoo Biscoe #> 4 2 2 4725 Gentoo Biscoe #> 5 3 1 4000 Chinstrap Dream #> 6 3 2 4050 Chinstrap Dream There are many more developments related to `*_join()` functions (e.g., [inequality joins](/blog/2023/01/dplyr-1-1-0-joins/#inequality-joins) and [rolling joins](/blog/2023/01/dplyr-1-1-0-joins/#rolling-joins) ), but many of these likely wouldn’t come up in an introductory course so we won’t get into their details. A good place to read more about them is [R for Data Science, 2nd edition](https://r4ds.hadley.nz/joins.html#sec-non-equi-joins) . Exploding joins (i.e., joins that result in a larger number of rows than either of the data frames from bie) can be hard to debug for students! Teaching them the tools to diagnose whether the join they performed, and that may not have given an error, is indeed the one they wanted to perform. Did they lose any cases? Did they gain an unexpected amount of cases? Did they perform a join without thinking and take down the entire teaching server? These things happen, particularly if students are working with their own data for an open-ended project! Per operation grouping[](#per-operation-grouping) -------------------------------------------------- To calculate grouped summary statistics, you previously needed to do something like this: # previously df |> group_by(x) |> summarize(mean(y)) Now, an alternative approach is to pass the groups directly in the [`summarize()`](https://dplyr.tidyverse.org/reference/summarise.html) call: # now, optionally df |> summarize( mean(y), .by = x ) Let’s take a look at the differences between these two approaches before making a recommendation for one over the other. [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) can result in groups that persist in the output, particularly when grouping by multiple variables. For example, in the following pipeline we group the penguins data frame by `species` and `sex`, find mean body weights for each resulting species / sex combination, and then show the first observation in the output with `slice_head(n = 1)`. Since the output is grouped by species, this results in one summary statistic per species. penguins |> drop_na(sex, body_mass_g) |> group_by(species, sex) |> summarize(mean_bw = mean(body_mass_g)) |> slice_head(n = 1) #> `summarise()` has grouped output by 'species'. You can override using the #> `.groups` argument. #> # A tibble: 3 × 3 #> # Groups: species [3] #> species sex mean_bw #> #> 1 Adelie female 3369. #> 2 Chinstrap female 3527. #> 3 Gentoo female 4680. If we explicitly drop the groups in the [`summarize()`](https://dplyr.tidyverse.org/reference/summarise.html) call, so that the output is no longer grouped, we get just one row in our output. penguins |> drop_na(sex, body_mass_g) |> group_by(species, sex) |> summarize(mean_bw = mean(body_mass_g), .groups = "drop") |> slice_head(n = 1) #> # A tibble: 1 × 3 #> species sex mean_bw #> #> 1 Adelie female 3369. This pair of examples show that whether your output is grouped or not can affect downstream results, and if you’re a [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) user, you’ve probably been burnt by this once or twice. Per-operation grouping allows you to define groups in a `.by` argument, and these groups don’t persist. So, regardless of whether you group by one or two variables, the resulting data frame after calculating a summary statistic is not grouped. # group by 1 variable penguins |> drop_na(sex, body_mass_g) |> summarize( mean_bw = mean(body_mass_g), .by = species ) #> # A tibble: 3 × 2 #> species mean_bw #> #> 1 Adelie 3706. #> 2 Gentoo 5092. #> 3 Chinstrap 3733. # group by 2 variables penguins |> drop_na(sex, body_mass_g) |> summarize( mean_bw = mean(body_mass_g), .by = c(species, sex) ) #> # A tibble: 6 × 3 #> species sex mean_bw #> #> 1 Adelie male 4043. #> 2 Adelie female 3369. #> 3 Gentoo female 4680. #> 4 Gentoo male 5485. #> 5 Chinstrap female 3527. #> 6 Chinstrap male 3939. So, when teaching grouped operations, you now have the option to choose between these two approaches. The most important teaching tip I can give, particularly for teaching to new learners, is to choose one method and stick to it. The `.by` method will result in fewer outputs that are unintentionally grouped, and hence, might potentially be easier for new learners. And while this approach is mentioned in R for Data Science, 2nd edition, the [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) approach is described in more detail. On the other hand. for more experienced learners, particularly those learning to design their own functions and packages, the evolution of grouping in the tidyverse can be an interesting subject to review. Quality of life improvements to `case_when()` and `if_else()`[](#quality-of-life-improvements-to-case_when-and-if_else) ------------------------------------------------------------------------------------------------------------------------ ### `case_when()`[](#case_when) Previously, when writing a [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) statement, you had to use `TRUE` to indicate “all else”. Additionally, [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) has historically been strict about the types on the right-hand side, e.g., requiring `NA_character` when other right-hand side values are characters, and not letting you get away with just `NA`. # previously df |> mutate( x = case_when( ~ "value 1", ~ "value 2", ~ "value 3", TRUE ~ NA_character_ ) ) Now, optionally, you can define “all else” in a `.default` argument of [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) and you no longer need to worry about the type of `NA` you use on the right-hand side. # now, optionally df |> mutate( x = case_when( ~ "value 1", ~ "value 2", ~ "value 3", .default = NA ) ) For example, you can now do something like the following when creating a categorical version of a numerical variable that has some `NA`s. penguins |> mutate( bm_cat = case_when( is.na(body_mass_g) ~ NA, body_mass_g < 3550 ~ "Small", between(body_mass_g, 3550, 4750) ~ "Medium", .default = "Large" ) ) |> relocate(body_mass_g, bm_cat) #> # A tibble: 344 × 9 #> body_mass_g bm_cat species island bill_length_mm bill_depth_mm #> #> 1 3750 Medium Adelie Torgersen 39.1 18.7 #> 2 3800 Medium Adelie Torgersen 39.5 17.4 #> 3 3250 Small Adelie Torgersen 40.3 18 #> 4 NA NA Adelie Torgersen NA NA #> 5 3450 Small Adelie Torgersen 36.7 19.3 #> 6 3650 Medium Adelie Torgersen 39.3 20.6 #> 7 3625 Medium Adelie Torgersen 38.9 17.8 #> 8 4675 Medium Adelie Torgersen 39.2 19.6 #> 9 3475 Small Adelie Torgersen 34.1 18.1 #> 10 4250 Medium Adelie Torgersen 42 20.2 #> # ℹ 334 more rows #> # ℹ 3 more variables: flipper_length_mm , sex , year ### `if_else()`[](#if_else) Similarly, [`if_else()`](https://dplyr.tidyverse.org/reference/if_else.html) is no longer as strict about typed missing values either. penguins |> mutate( bm_unit = if_else(!is.na(body_mass_g), paste(body_mass_g, "g"), NA) ) |> relocate(body_mass_g, bm_unit) #> # A tibble: 344 × 9 #> body_mass_g bm_unit species island bill_length_mm bill_depth_mm #> #> 1 3750 3750 g Adelie Torgersen 39.1 18.7 #> 2 3800 3800 g Adelie Torgersen 39.5 17.4 #> 3 3250 3250 g Adelie Torgersen 40.3 18 #> 4 NA NA Adelie Torgersen NA NA #> 5 3450 3450 g Adelie Torgersen 36.7 19.3 #> 6 3650 3650 g Adelie Torgersen 39.3 20.6 #> 7 3625 3625 g Adelie Torgersen 38.9 17.8 #> 8 4675 4675 g Adelie Torgersen 39.2 19.6 #> 9 3475 3475 g Adelie Torgersen 34.1 18.1 #> 10 4250 4250 g Adelie Torgersen 42 20.2 #> # ℹ 334 more rows #> # ℹ 3 more variables: flipper_length_mm , sex , year While these may be seemingly small improvements, I think they have huge benefits for teaching and learning. It’s a blessing to not have to introduce `NA_character_` and friends as early as introducing [`if_else()`](https://dplyr.tidyverse.org/reference/if_else.html) and [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) ! Different types of `NA`s are a good topic for a course on R as a programming language, statistical computing, etc. but they are unnecessarily complex for an introductory course. New syntax for separating columns[](#new-syntax-for-separating-columns) ------------------------------------------------------------------------ The following table summarizes new syntax for separating columns in tidyr that supersede [`extract()`](https://tidyr.tidyverse.org/reference/extract.html) , [`separate()`](https://tidyr.tidyverse.org/reference/separate.html) , and [`separate_rows()`](https://tidyr.tidyverse.org/reference/separate_rows.html) . These updates are motivated by the goal of achieving a set of functions that have more consistent names and arguments, have better performance, and provide a new approach for handling problems: | | **MAKE COLUMNS** | **MAKE ROWS** | | --- | --- | --- | | Separate with delimiter | [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) | [`separate_longer_delim()`](https://tidyr.tidyverse.org/reference/separate_longer_delim.html) | | Separate by position | [`separate_wider_position()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) | [`separate_longer_position()`](https://tidyr.tidyverse.org/reference/separate_longer_delim.html) | | Separate with regular expression | | | Here is an example for using some of these functions. Let’s suppose we have data on three penguins with their descriptions. three_penguin_descriptions <- tribble( ~id, ~description, 1, "Species: Adelie, Island - Torgersen", 2, "Species: Gentoo, Island - Biscoe", 3, "Species: Chinstrap, Island - Dream", ) three_penguin_descriptions #> # A tibble: 3 × 2 #> id description #> #> 1 1 Species: Adelie, Island - Torgersen #> 2 2 Species: Gentoo, Island - Biscoe #> 3 3 Species: Chinstrap, Island - Dream We can seaprate the description column into `species` and `island` with [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) : three_penguin_descriptions |> separate_wider_delim( cols = description, delim = ", ", names = c("species", "island") ) #> # A tibble: 3 × 3 #> id species island #> #> 1 1 Species: Adelie Island - Torgersen #> 2 2 Species: Gentoo Island - Biscoe #> 3 3 Species: Chinstrap Island - Dream Or we can do so with regular expressions: three_penguin_descriptions |> separate_wider_regex( cols = description, patterns = c( "Species: ", species = "[^,]+", ", ", "Island - ", island = ".*" ) ) #> # A tibble: 3 × 3 #> id species island #> #> 1 1 Adelie Torgersen #> 2 2 Gentoo Biscoe #> 3 3 Chinstrap Dream If teaching folks coming from doing data manipulation in spreadsheets, leverage that to motivate different types of `separate_*()` functions, and show the benefits of programming over point-and-click software for more advanced operations like separating longer and separating with regular expressions. New argument for line geoms: `linewidth`[](#new-argument-for-line-geoms-linewidth) ----------------------------------------------------------------------------------- If you, like me, have a bunch of scatterplots with smooth lines overlaid on them, you might run into the following warning. # previously penguins |> drop_na() |> ggplot(aes(x = flipper_length_mm, y = body_mass_g)) + geom_smooth(size = 2) #> Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0. #> ℹ Please use `linewidth` instead. #> This warning is displayed once every 8 hours. #> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was #> generated. #> `geom_smooth()` using method = 'loess' and formula = 'y ~ x' ![](figs/unnamed-chunk-42-1.png) Instead of `size`, you should now be using `linewidth`. # now penguins |> drop_na() |> ggplot(aes(x = flipper_length_mm, y = body_mass_g)) + geom_smooth(linewidth = 2) #> `geom_smooth()` using method = 'loess' and formula = 'y ~ x' ![](figs/unnamed-chunk-43-1.png) The teaching tip should be obvious here… Check the output of your old teaching materials thoroughly to not make a fool of yourself when teaching! 🤣 Other highlights[](#other-highlights) -------------------------------------- * purrr 1.0.0: While purrr is likely not a common topic in introductory data science curricula, if you do teach iteration with purrr, you’ll want to check out the [purrr 1.0.0 blog post](https://www.tidyverse.org/blog/2022/12/purrr-1-0-0/) . I also highly recommend [Hadley’s purrr video](https://youtu.be/EGAs7zuRutY) to those who are new to purrr as well as those who want to quickly review most recent updates to it. * webR 0.1.0: webR provides a framework for creating websites where users can run R code directly within the web browser, without R installed on their device or a supporting computational R server. This is hugely exciting for writing educational materials, like interactive lesson notes, and there’s already a Quarto extension that allows you to do this: [https://github.com/coatless/quarto-webr](https://github.com/coatless/quarto-webr) . I think this is an important space to watch for educators! Coming up[](#coming-up) ------------------------ I will be teaching a “Teaching Data Science Masterclass” at posit::conf(2023), with a module specifically on teaching the Tidyverse. [Watch the course trailer](https://youtu.be/5TVd_whxUus) and [read the full course description](https://reg.conf.posit.co/flow/posit/positconf23/attendee-portal/page/sessioncatalog?search=%22Teaching%20Data%20Science%20Masterclass%22&search.sessiontype=1675316728702001wr6r) if you’d like to find out more and sign up! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # fs [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Fs [![](/blog/2023/05/purrr-walk-this-way/thumbnail-sq.jpg)](/blog/2023/05/purrr-walk-this-way/) [`purrr::walk()` this way](/blog/2023/05/purrr-walk-this-way/) [learn](/categories/learn) Mara Averick How to use `purrr::walk()` to write many files, featuring file-system navigation with the fs package. [Read more ...](/blog/2023/05/purrr-walk-this-way/) 2023/05/26 [![](/blog/2018/01/fs-1.0.0/fs-1.0.0-sq.jpg)](/blog/2018/01/fs-1.0.0/) [fs 1.0.0](/blog/2018/01/fs-1.0.0/) [package](/categories/package) Jim Hester fs 1.0.0 is now available on CRAN. fs provides a cross-platform, uniform interface to file system operations. [Read more ...](/blog/2018/01/fs-1.0.0/) 2018/01/22 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q1 2023 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q1 2023 tidymodels digest ========================= ![](/blog/2023/04/tidymodels-2023-q1/thumbnail-wd.jpg) Photo by [Chi Liu](https://unsplash.com/photos/l-rtCtc_4c0)   2023/04/28   [tidymodels](/tags/tidymodels/) , [recipes](/tags/recipes/) , [yardstick](/tags/yardstick/) , [dials](/tags/dials/)   Emil Hvitfeldt The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like these posts from the past couple of months: * [Tuning hyperparameters with tidymodels is a delight](https://www.tidyverse.org/blog/2023/04/tuning-delights/) * [censored 0.2.0](https://www.tidyverse.org/blog/2023/04/censored-0-2-0/) * [The tidymodels is getting a whole lot faster](https://www.simonpcouch.com/blog/speedups-2023/) Since [our last roundup post](https://www.tidyverse.org/blog/2022/12/tidymodels-2022-q4/) , there have been CRAN releases of 24 tidymodels packages. Here are links to their NEWS files: * agua [(0.1.2)](https://agua.tidymodels.org/news/index.html) * baguette [(1.0.1)](https://baguette.tidymodels.org/news/index.html) * broom [(1.0.4)](https://broom.tidymodels.org/news/index.html) * butcher [(0.3.2)](https://butcher.tidymodels.org/news/index.html) * censored [(0.2.0)](https://censored.tidymodels.org/news/index.html) * dials [(1.2.0)](https://dials.tidymodels.org/news/index.html) * discrim [(1.0.1)](https://discrim.tidymodels.org/news/index.html) * embed [(1.1.0)](https://embed.tidymodels.org/news/index.html) * finetune [(1.1.0)](https://finetune.tidymodels.org/news/index.html) * hardhat [(1.3.0)](https://hardhat.tidymodels.org/news/index.html) * modeldata [(1.1.0)](https://modeldata.tidymodels.org/news/index.html) * parsnip [(1.1.0)](https://parsnip.tidymodels.org/news/index.html) * recipes [(1.0.6)](https://recipes.tidymodels.org/news/index.html) * rules [(1.0.2)](https://rules.tidymodels.org/news/index.html) * spatialsample [(0.3.0)](https://spatialsample.tidymodels.org/news/index.html) * stacks [(1.0.2)](https://stacks.tidymodels.org/news/index.html) * textrecipes [(1.0.3)](https://textrecipes.tidymodels.org/news/index.html) * themis [(1.0.1)](https://themis.tidymodels.org/news/index.html) * tidyclust [(0.1.2)](https://tidyclust.tidymodels.org/news/index.html) * tidypredict [(0.5)](https://tidypredict.tidymodels.org/news/index.html) * tune [(1.1.1)](https://tune.tidymodels.org/news/index.html) * workflows [(1.1.3)](https://workflows.tidymodels.org/news/index.html) * workflowsets [(1.0.1)](https://workflowsets.tidymodels.org/news/index.html) * yardstick [(1.2.0)](https://yardstick.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: more informative errors and faster code. First, loading the collection of packages: library(tidymodels) library(embed) data("ames", package = "modeldata") More informative errors[](#more-informative-errors) ---------------------------------------------------- In the last few months we have been focused on refining error messages so that they are easier for the users to pinpoint what went wrong and where. Since the modeling pipeline can be quite complicated, getting uninformative errors is a no-go. Across the tidymodels, error messages will now indicate the user-facing function that caused the error rather than the internal function that it came from. From dials, an error that looked like degree(range = c(1L, 5L)) #> Error in `new_quant_param()`: #> ! Since `type = 'double'`, please use that data type for the range. Now says that the error came from `degree()` rather than `new_quant_param()` degree(range = c(1L, 5L)) #> Error in `degree()`: #> ! Since `type = 'double'`, please use that data type for the range. The same thing can be seen with the yardstick metrics mtcars |> accuracy(vs, am) #> Error in `dplyr::summarise()`: #> ℹ In argument: `.estimate = metric_fn(truth = vs, estimate = am, na_rm = #> na_rm)`. #> Caused by error in `validate_class()`: #> ! `truth` should be a factor but a numeric was supplied. which now errors much more informatively mtcars |> accuracy(vs, am) #> Error in `accuracy()`: #> ! `truth` should be a factor, not a `numeric`. Lastly, one of the biggest improvements came in recipes, which now shows which step caused the error instead of saying it happened in [`prep()`](https://recipes.tidymodels.org/reference/prep.html) or [`bake()`](https://recipes.tidymodels.org/reference/bake.html) . This is a huge improvement since preprocessing pipelines which often string together many preprocessing steps. Before recipe(~., data = ames) |> step_novel(Neighborhood, new_level = "Gilbert") |> prep() #> Error in `prep()`: #> ! Columns already contain the new level: Neighborhood Now recipe(~., data = ames) |> step_novel(Neighborhood, new_level = "Gilbert") |> prep() #> Error in `step_novel()`: #> Caused by error in `prep()` at recipes/R/recipe.R:437:8: #> ! Columns already contain the new level: Neighborhood Especially when calls to recipes functions are deeply nested inside the call stack, like in `fit_resamples()` or `tune_grid()`, these changes make a big difference. Things are getting faster[](#things-are-getting-faster) -------------------------------------------------------- As we have written about in [The tidymodels is getting a whole lot faster](https://www.simonpcouch.com/blog/speedups-2023/) and [Writing performant code with tidy tools](https://www.tidyverse.org/blog/2023/04/performant-packages/) , we have been working on tightening up the performance of the tidymodels code. These changes are mostly related to the infrastructure code, meaning that the speedup will bring you to closer underlying implementations. A different kind of speedup is found with the addition of the [step\_pca\_truncated()](https://embed.tidymodels.org/reference/step_pca_truncated.html) step added in the embed package. [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) is a really powerful and fast method for dimensionality reduction of large data sets. However, for data with many columns, it can be computationally expensive to calculate all the principal components. [`step_pca_truncated()`](https://embed.tidymodels.org/reference/step_pca_truncated.html) works in much the same way as [`step_pca()`](https://recipes.tidymodels.org/reference/step_pca.html) but it only calculates the number of components it needs pca_normal <- recipe(Sale_Price ~ ., data = ames) |> step_dummy(all_nominal_predictors()) |> step_pca(all_numeric_predictors(), num_comp = 3) pca_truncated <- recipe(Sale_Price ~ ., data = ames) |> step_dummy(all_nominal_predictors()) |> step_pca_truncated(all_numeric_predictors(), num_comp = 3) tictoc::tic() prep(pca_normal) |> bake(ames) #> # A tibble: 2,930 × 4 #> Sale_Price PC1 PC2 PC3 #> #> 1 215000 -31793. 4151. -197. #> 2 105000 -12198. -611. -524. #> 3 172000 -14911. -265. 7568. #> 4 244000 -12072. -1813. 918. #> 5 189900 -14418. -345. -302. #> 6 195500 -10704. -1367. -204. #> 7 213500 -5858. -2805. 114. #> 8 191500 -5932. -2762. 131. #> 9 236500 -6368. -2862. 325. #> 10 189000 -8368. -2219. 126. #> # ℹ 2,920 more rows tictoc::toc() #> 0.782 sec elapsed tictoc::tic() prep(pca_truncated) |> bake(ames) #> # A tibble: 2,930 × 4 #> Sale_Price PC1 PC2 PC3 #> #> 1 215000 -31793. 4151. -197. #> 2 105000 -12198. -611. -524. #> 3 172000 -14911. -265. 7568. #> 4 244000 -12072. -1813. 918. #> 5 189900 -14418. -345. -302. #> 6 195500 -10704. -1367. -204. #> 7 213500 -5858. -2805. 114. #> 8 191500 -5932. -2762. 131. #> 9 236500 -6368. -2862. 325. #> 10 189000 -8368. -2219. 126. #> # ℹ 2,920 more rows tictoc::toc() #> 0.162 sec elapsed The speedup will be orders of magnitude larger for very wide data. Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to thank those in the community that contributed to tidymodels in the last quarter: * agua: [@hfrick](https://github.com/hfrick) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * baguette: [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * broom: [@benwhalley](https://github.com/benwhalley) , [@dgrtwo](https://github.com/dgrtwo) , [@egosv](https://github.com/egosv) , [@hfrick](https://github.com/hfrick) , [@JorisChau](https://github.com/JorisChau) , [@mccarthy-m-g](https://github.com/mccarthy-m-g) , [@MichaelChirico](https://github.com/MichaelChirico) , [@paige-cho](https://github.com/paige-cho) , [@PoGibas](https://github.com/PoGibas) , [@rsbivand](https://github.com/rsbivand) , [@simonpcouch](https://github.com/simonpcouch) , [@ste-tuf](https://github.com/ste-tuf) , and [@victor-vscn](https://github.com/victor-vscn) . * butcher: [@ashbythorpe](https://github.com/ashbythorpe) , [@DavisVaughan](https://github.com/DavisVaughan) , [@hfrick](https://github.com/hfrick) , [@juliasilge](https://github.com/juliasilge) , [@rdavis120](https://github.com/rdavis120) , [@rkb965](https://github.com/rkb965) , and [@simonpcouch](https://github.com/simonpcouch) . * censored: [@brunocarlin](https://github.com/brunocarlin) , and [@hfrick](https://github.com/hfrick) . * dials: [@amin0511ss](https://github.com/amin0511ss) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , and [@simonpcouch](https://github.com/simonpcouch) . * discrim: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@tomwagstaff-opml](https://github.com/tomwagstaff-opml) . * embed: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@jackobenco016](https://github.com/jackobenco016) , and [@skasowitz](https://github.com/skasowitz) . * finetune: [@Freestyleyang](https://github.com/Freestyleyang) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * hardhat: [@cregouby](https://github.com/cregouby) , [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@frank113](https://github.com/frank113) , and [@mikemahoney218](https://github.com/mikemahoney218) . * modeldata: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , and [@topepo](https://github.com/topepo) . * parsnip: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@emmafeuer](https://github.com/emmafeuer) , [@exsell-jc](https://github.com/exsell-jc) , [@hfrick](https://github.com/hfrick) , [@mariamaseng](https://github.com/mariamaseng) , [@SHo-JANG](https://github.com/SHo-JANG) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@Tripartio](https://github.com/Tripartio) . * recipes: [@AshesITR](https://github.com/AshesITR) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@jjcurtin](https://github.com/jjcurtin) , [@lang-benjamin](https://github.com/lang-benjamin) , [@lbui30](https://github.com/lbui30) , [@PeterKoffeldt](https://github.com/PeterKoffeldt) , [@rdavis120](https://github.com/rdavis120) , [@simonpcouch](https://github.com/simonpcouch) , [@StevenWallaert](https://github.com/StevenWallaert) , [@tellyshia](https://github.com/tellyshia) , [@topepo](https://github.com/topepo) , [@ttrodrigz](https://github.com/ttrodrigz) , and [@zecojls](https://github.com/zecojls) . * rules: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@jonthegeek](https://github.com/jonthegeek) , and [@topepo](https://github.com/topepo) . * spatialsample: [@hfrick](https://github.com/hfrick) , [@mikemahoney218](https://github.com/mikemahoney218) , and [@RaymondBalise](https://github.com/RaymondBalise) . * stacks: [@amin0511ss](https://github.com/amin0511ss) , [@gundalav](https://github.com/gundalav) , [@jrosell](https://github.com/jrosell) , [@juliasilge](https://github.com/juliasilge) , [@pbulsink](https://github.com/pbulsink) , [@rdavis120](https://github.com/rdavis120) , and [@simonpcouch](https://github.com/simonpcouch) . * textrecipes: [@apsteinmetz](https://github.com/apsteinmetz) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gary-mu](https://github.com/gary-mu) , [@hfrick](https://github.com/hfrick) , and [@nipnipj](https://github.com/nipnipj) . * themis: [@carlganz](https://github.com/carlganz) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@nipnipj](https://github.com/nipnipj) , [@rmurphy49](https://github.com/rmurphy49) , and [@rowanjh](https://github.com/rowanjh) . * tidyclust: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@hsbadr](https://github.com/hsbadr) , [@jonthegeek](https://github.com/jonthegeek) , and [@simonpcouch](https://github.com/simonpcouch) . * tidypredict: [@edgararuiz](https://github.com/edgararuiz) , and [@sdcharle](https://github.com/sdcharle) . * tune: [@BenoitLondon](https://github.com/BenoitLondon) , [@cphaarmeyer](https://github.com/cphaarmeyer) , [@hfrick](https://github.com/hfrick) , [@jthomasmock](https://github.com/jthomasmock) , [@mrjujas](https://github.com/mrjujas) , [@MxNl](https://github.com/MxNl) , [@nabsiddiqui](https://github.com/nabsiddiqui) , [@rdavis120](https://github.com/rdavis120) , [@SHo-JANG](https://github.com/SHo-JANG) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , [@walrossker](https://github.com/walrossker) , and [@yusuftengriverdi](https://github.com/yusuftengriverdi) . * workflows: [@simonpcouch](https://github.com/simonpcouch) . * workflowsets: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gsimchoni](https://github.com/gsimchoni) , and [@simonpcouch](https://github.com/simonpcouch) . * yardstick: [@77makr](https://github.com/77makr) , [@burch-cm](https://github.com/burch-cm) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@idavydov](https://github.com/idavydov) , [@kadyb](https://github.com/kadyb) , [@mawardivaz](https://github.com/mawardivaz) , [@mikemahoney218](https://github.com/mikemahoney218) , [@moloscripts](https://github.com/moloscripts) , [@nyambea](https://github.com/nyambea) , [@SHo-JANG](https://github.com/SHo-JANG) , [@simdadim](https://github.com/simdadim) , and [@simonpcouch](https://github.com/simonpcouch) . We’re grateful for all of the tidymodels community, from observers to users to contributors. Happy modeling! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # package [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Package [![](/blog/2023/04/performant-packages/thumbnail-sq.jpg)](/blog/2023/04/performant-packages/) [Writing performant code with tidy tools](/blog/2023/04/performant-packages/) [programming](/categories/programming) Simon Couch When performance becomes an issue for code using tidy interfaces, switching to the backend tools used by tidy developers can offer substantial speedups. [Read more ...](/blog/2023/04/performant-packages/) 2023/04/18 [![](/blog/2020/02/glue-strings-and-tidy-eval/glue-strings-and-tidy-eval-sq.jpg)](/blog/2020/02/glue-strings-and-tidy-eval/) [Tidy eval now supports glue strings](/blog/2020/02/glue-strings-and-tidy-eval/) [package](/categories/package) Lionel Henry [Read more ...](/blog/2020/02/glue-strings-and-tidy-eval/) 2020/02/11 [![](/blog/2020/01/vroom-1-1-0/thumbnail-sq.jpg)](/blog/2020/01/vroom-1-1-0/) [vroom 1.1.0](/blog/2020/01/vroom-1-1-0/) [package](/categories/package) Jim Hester vroom 1.1.0 is now on CRAN! [Read more ...](/blog/2020/01/vroom-1-1-0/) 2020/01/15 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Differences between the base R and magrittr pipes [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Differences between the base R and magrittr pipes ================================================= ![](/blog/2023/04/base-vs-magrittr-pipe/thumbnail-wd.jpg) Photo by [Sigmund](https://unsplash.com/photos/4CNNH2KEjhc)   2023/04/21   [magrittr](/tags/magrittr/)   Hadley Wickham **Note:** The following has been adapted from a section of the forthcoming second edition of [R for Data Science](https://r4ds.hadley.nz/) that had to be removed due to length limitations. Pipes[](#pipes) ---------------- R 4.1.0 introduced a native pipe operator, `|>`. As described in the [R News](https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) : > R now provides a simple native forward pipe syntax `|>`. The simple form of the forward pipe inserts the left-hand side as the first argument in the right-hand side call. The pipe implementation as a syntax transformation was motivated by suggestions from Jim Hester and Lionel Henry. The behaviour of the native pipe is by and large the same as that of the [`%>%`](https://magrittr.tidyverse.org/reference/pipe.html) pipe provided by the **magrittr** package. Both operators (`|>` and `%>%`) let you “pipe” an object forward to a function or call expression, thereby allowing you to express a sequence of operations that transform an object. To learn more about the basic utility of pipes, see [The pipe](https://r4ds.hadley.nz/data-transform.html#the-pipe) section of R for Data Science. Luckily there’s no need to commit entirely to one pipe or the other — you can use the base pipe for the majority of cases where it’s sufficient and use the magrittr pipe when you really need its special features. `|>` vs. `%>%`[](#-vs) ----------------------- While `|>` and `%>%` behave identically for simple cases, there are a few crucial differences. These are most likely to affect you if you’re a long-term user of `%>%` who has taken advantage of some of the more advanced features. But they’re still good to know about even if you’ve never used `%>%` because you’re likely to encounter some of them when reading wild-caught code. * By default, the pipe passes the object on its left-hand side to the first argument of the function on the right-hand side. `%>%` allows you to change the placement with a `.` placeholder. For example, `x %>% f(1)` is equivalent to `f(x, 1)` but `x %>% f(1, .)` is equivalent to `f(1, x)`. R 4.2.0 added a `_` placeholder to the base pipe, with one additional restriction: the argument has to be named. For example, `x |> f(1, y = _)` is equivalent to `f(1, y = x)`. * The `|>` placeholder is deliberately simple and can’t replicate many features of the `%>%` placeholder: you can’t pass it to multiple arguments, and it doesn’t have any special behavior when the placeholder is used inside another function. For example, `df %>% split(.$var)` is equivalent to `split(df, df$var)`, and `df %>% {split(.$x, .$y)}` is equivalent to `split(df$x, df$y)`. With `%>%`, you can use `.` on the left-hand side of operators like `$`, `[[`, `[` , so you can extract a single column from a data frame with (e.g.) `mtcars %>% .$cyl`. R added support for this feature in R 4.3.0. For the special case of extracting a column out of a data frame, you can also use `dplyr::pull()`:\ \ mtcars |> pull(cyl)\ \ * `%>%` allows you to drop the parentheses when calling a function with no other arguments; `|>` always requires the parentheses.\ \ * `%>%` allows you to start a pipe with `.` to create a function rather than immediately executing the pipe; this is not supported by the base pipe.\ \ \ Using the native pipe in packages[](#using-the-native-pipe-in-packages)\ \ ------------------------------------------------------------------------\ \ Because the native pipe wasn’t introduced until 4.1.0, code using `|>` in function reference examples or vignettes will not work on older versions of R, as it is not valid syntax. This is a problem for the tidyverse because our [versioning policies](https://www.tidyverse.org/blog/2019/04/r-version-support/)\ mean that our packages need to work on R 3.5.0 and later.\ \ Does this mean that you need to increase the minimum R version your package depends on in order to use `|>`? Not necessarily: there are two techniques we can use to keep vignettes and examples working.\ \ For example, the base pipe is used in purrr 1.0.0. As can be seen in the [source for the “purrr <-> base R” vignette](https://github.com/tidyverse/purrr/commit/df4630c6e8cd5028386ee96b9036f1755f26adc4)\ , certain code chunks are evaluated conditionally based on the version of R being used. The setup chunk for the vignette includes: `modern_r <- getRversion() >= "4.1.0"`. The results of this are then used in the `eval` argument to determine whether or not a code chunk that relies on “modern R” syntax should be run.\ \ The other place we use the base pipe is in examples. To disable these we use a bit of a hack that requires three files [`configure`](https://github.com/tidyverse/purrr/blob/main/configure)\ , [`cleanup`](https://github.com/tidyverse/purrr/blob/main/cleanup)\ , and [`tools/examples.R`](https://github.com/tidyverse/purrr/blob/main/tools/examples.R)\ . The basic idea is for pre-R 4.1.0 we re-define the `\examples{}` tag to display an informative message but not run the code; this ensures that `R CMD check` continues to work even on older versions of R.\ \ Contents\ \ The tidyverse is proudly supported by[](https://posit.co/)\ \ [Privacy policy](https://www.tidyverse.org/google_privacy_policy)\ [](https://github.com/tidyverse)\ [](https://twitter.com/hashtag/rstats) --- # censored 0.2.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) censored 0.2.0 ============== ![](/blog/2023/04/censored-0-2-0/thumbnail-wd.jpg) Photo by [Sam Poullain](https://unsplash.com/photos/TuAZPj1uaZs)   2023/04/19   [tidymodels](/tags/tidymodels/) , [parsnip](/tags/parsnip/) , [censored](/tags/censored/)   Hannah Frick We’re thrilled to announce the release of [censored](https://censored.tidymodels.org/) 0.2.0. censored is a parsnip extension package for survival models. You can install it from CRAN with: install.packages("censored") This blog post will introduce you to a new argument name, `eval_time`, and two new engines for fitting random forests and parametric survival models. You can see a full list of changes in the [release notes](https://github.com/tidymodels/censored/releases/tag/v0.2.0) . Introducing `eval_time`[](#introducing-eval_time) -------------------------------------------------- As we continue to add support for survival analysis across tidymodels, we have seen a need to be more explicit about which time we mean when we say “time”: event time, observed time, censoring time, time at which to predict survival probability at? The last one is a particular mouthful. We now refer to this time as “evaluation time.” In preparation for dynamic survival performance metrics which can be calculated at different evaluation time points, the argument to set these evaluation time points for [`predict()`](https://rdrr.io/r/stats/predict.html) is now called `eval_time` instead of just `time`. cox <- proportional_hazards() |> set_engine("survival") |> set_mode("censored regression") |> fit(Surv(time, status) ~ ., data = lung) pred <- predict(cox, lung[1:3, ], type = "survival", eval_time = c(100, 500)) pred #> # A tibble: 3 × 1 #> .pred #> #> 1 #> 2 #> 3 The predictions follow the tidymodels principle of one row per observation, and the nested tibble contains the predicted survival probability, `.pred_survival`, as well as the corresponding evaluation time. The column for the evaluation time is now called `.eval_time` instead of `.time`. pred$.pred[[2]] #> # A tibble: 2 × 2 #> .eval_time .pred_survival #> #> 1 100 0.910 #> 2 500 0.422 New engines[](#new-engines) ---------------------------- censored contains engines for parametric, semi-parametric, and tree-based models. This release adds two new engines: * the `"aorsf"` engine for random forests via [`rand_forest()`](https://parsnip.tidymodels.org/reference/rand_forest.html) * the `"flexsurvspline"` engine for parametric models via [`survival_reg()`](https://parsnip.tidymodels.org/reference/survival_reg.html) ### New `"aorsf"` engine for `rand_forest()`[](#new-aorsf-engine-for-rand_forest) This engine has been contributed by [Byron Jaeger](https://github.com/bcjaeger) and enables users to fit oblique random survival forests with the aorsf package. What’s with the _oblique_ you ask? Oblique describes how the decision trees that form the random forest make their splits at each node. If the split is based on a single predictor, the resulting tree is called _axis-based_ because the split is perpendicular to the axis of the predictor. If the split is based on a linear combination of predictors, there is a lot more flexibility in how the data is split: the split does not need to be perpendicular to any of the predictor axes. Such trees are called _oblique_. The documentation for the [aorsf](https://docs.ropensci.org/aorsf) package includes a nice illustration of this with the splits for an axis-based tree on the left and an oblique tree on the right: ![Two scatter plots of data with two predictors, X1 and X2, and two classes, coded as pink dots and orange squares. The lefthand plot shows the splits of an axis-based decision tree which are at a right angle to the axis. The resulting partition generally separates the classes well but not perfectly. The righthand plot shows the splits of an oblique tree which achieves perfect separation on this example because it can cut across the predictor space diagnonally.](https://docs.ropensci.org/aorsf/reference/figures/tree_axis_v_oblique.png) To fit such a model, set the engine for a random forest to `"aorsf"`: lung <- na.omit(lung) forest <- rand_forest() |> set_engine("aorsf") |> set_mode("censored regression") |> fit(Surv(time, status) ~ ., data = lung) pred <- predict(forest, lung[1:3, ], type = "survival", eval_time = c(100, 500)) pred$.pred[[1]] #> # A tibble: 2 × 2 #> .eval_time .pred_survival #> #> 1 100 0.928 #> 2 500 0.368 ### New `"flexsurvspline"` engine for `survival_reg()`[](#new-flexsurvspline-engine-for-survival_reg) This engine has been contributed by [Matt Warkentin](https://github.com/mattwarkentin) and enables users to fit a parametric survival model with splines via [`flexsurv::flexsurvspline()`](https://rdrr.io/pkg/flexsurv/man/flexsurvspline.html) . This model uses natural cubic splines to model a transformation of the survival function, e.g., the log cumulative hazard. This gives a lot more flexibility to a parametric model allowing us, for example, to represent more irregular hazard curves. Let’s illustrate that with a data set of survival times of breast cancer patients, based on the example from [Jackson (2016)](https://www.jstatsoft.org/article/view/v070i08) . The flexibility of the model is governed by `k`, the number of knots in the spline. We set `scale = "odds"` for a proportional hazards model. data(bc, package = "flexsurv") fit_splines <- survival_reg() |> set_engine("flexsurvspline", k = 5, scale = "odds") |> fit(Surv(recyrs, censrec) ~ group, data = bc) For comparison, we also fit a parametric model without splines. fit_gengamma <- survival_reg(dist = "gengamma") |> set_engine("flexsurv") |> fit(Surv(recyrs, censrec) ~ group, data = bc) We can predict the hazard for the three levels of the prognostic `group`. bc_groups <- tibble(group = c("Poor","Medium","Good")) pred_splines <- predict(fit_splines, new_data = bc_groups, type = "hazard", eval_time = seq(0.1, 8, by = 0.1)) |> mutate(model = "splines") |> bind_cols(bc_groups) pred_gengamma <- predict(fit_gengamma, new_data = bc_groups, type = "hazard", eval_time = seq(0.1, 8, by = 0.1)) |> mutate(model = "gengamma") |> bind_cols(bc_groups) Plotting the predictions of both models shows a lot more flexibility in the splines model. bind_rows(pred_splines, pred_gengamma) %>% mutate(group = factor(group, levels = c("Poor","Medium","Good"))) |> tidyr::unnest(cols = .pred) |> ggplot() + geom_line(aes(x = .eval_time, y = .pred_hazard, group = group, col = group)) + facet_wrap(~ model) ![Two panels side by side, showing the predicted hazard curves for the three prognostic groups from the parametric model on the left and the spline model on the right. The curves for the spline model show more wiggliness, having more flexibility to adapt to the data than the curves from the parametric model which have to follow a generalized gamma distribution.](figs/unnamed-chunk-8-1.png) Acknowledgements[](#acknowledgements) -------------------------------------- Special thanks to Matt Warkentin and Byron Jaeger for the new engines! A big thank you to all the people who have contributed to censored since the release of v0.1.0: [@bcjaeger](https://github.com/bcjaeger) , [@hfrick](https://github.com/hfrick) , [@mattwarkentin](https://github.com/mattwarkentin) , [@simonpcouch](https://github.com/simonpcouch) , [@therneau](https://github.com/therneau) , and [@topepo](https://github.com/topepo) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Tuning hyperparameters with tidymodels is a delight [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tuning hyperparameters with tidymodels is a delight =================================================== ![](/blog/2023/04/tuning-delights/thumbnail-wd.jpg) Photo by [Mario La Pergola](https://unsplash.com/photos/Wrx0iVcYKmM)   2023/04/20   [tidymodels](/tags/tidymodels/) , [workflowsets](/tags/workflowsets/) , [tune](/tags/tune/)   Simon Couch The tidymodels team recently released new versions of the tune, finetune, and workflowsets packages, and we’re super stoked about it! Each of these three packages facilitates tuning hyperparameters in tidymodels, and their new releases work to make the experience of hyperparameter tuning more joyful. You can install these releases from CRAN with: install.packages(c("tune", "workflowsets", "finetune")) This blog post will highlight some of new changes in these packages that we’re most excited about. You can see the full lists of changes in the release notes for each package: * [tune v1.1.0](https://github.com/tidymodels/tune/releases/tag/v1.1.0) * [workflowsets v1.0.1](https://github.com/tidymodels/workflowsets/releases/tag/v1.0.1) * [finetune v1.1.0](https://github.com/tidymodels/finetune/releases/tag/v1.1.0) library(tidymodels) library(finetune) A shorthand for fitting the optimal model[](#a-shorthand-for-fitting-the-optimal-model) ---------------------------------------------------------------------------------------- In tidymodels, the result of tuning a set of hyperparameters is a data structure describing the candidate models, their predictions, and the performance metrics associated with those predictions. For example, tuning the number of `neighbors` in a `nearest_neighbors()` model over a regular grid: # tune the `neighbors` hyperparameter knn_model_spec <- nearest_neighbor("regression", neighbors = tune()) tuning_res <- tune_grid( knn_model_spec, mpg ~ ., bootstraps(mtcars, 5), control = control_grid(save_workflow = TRUE) ) # check out the resulting object tuning_res #> # Tuning results #> # Bootstrap sampling #> # A tibble: 5 × 4 #> splits id .metrics .notes #> #> 1 Bootstrap1 #> 2 Bootstrap2 #> 3 Bootstrap3 #> 4 Bootstrap4 #> 5 Bootstrap5 # examine proposed hyperparameters and associated metrics collect_metrics(tuning_res) #> # A tibble: 20 × 7 #> neighbors .metric .estimator mean n std_err .config #> #> 1 2 rmse standard 3.19 5 0.208 Preprocessor1_Model01 #> 2 2 rsq standard 0.664 5 0.0861 Preprocessor1_Model01 #> 3 3 rmse standard 3.13 5 0.266 Preprocessor1_Model02 #> 4 3 rsq standard 0.678 5 0.0868 Preprocessor1_Model02 #> 5 4 rmse standard 3.11 5 0.292 Preprocessor1_Model03 #> 6 4 rsq standard 0.684 5 0.0851 Preprocessor1_Model03 #> 7 5 rmse standard 3.10 5 0.287 Preprocessor1_Model04 #> 8 5 rsq standard 0.686 5 0.0839 Preprocessor1_Model04 #> 9 8 rmse standard 3.08 5 0.263 Preprocessor1_Model05 #> 10 8 rsq standard 0.689 5 0.0843 Preprocessor1_Model05 #> 11 9 rmse standard 3.07 5 0.256 Preprocessor1_Model06 #> 12 9 rsq standard 0.691 5 0.0845 Preprocessor1_Model06 #> 13 10 rmse standard 3.06 5 0.247 Preprocessor1_Model07 #> 14 10 rsq standard 0.693 5 0.0837 Preprocessor1_Model07 #> 15 11 rmse standard 3.05 5 0.241 Preprocessor1_Model08 #> 16 11 rsq standard 0.696 5 0.0833 Preprocessor1_Model08 #> 17 13 rmse standard 3.03 5 0.236 Preprocessor1_Model09 #> 18 13 rsq standard 0.701 5 0.0820 Preprocessor1_Model09 #> 19 14 rmse standard 3.02 5 0.235 Preprocessor1_Model10 #> 20 14 rsq standard 0.704 5 0.0808 Preprocessor1_Model10 Given these tuning results, the next steps are to choose the “best” hyperparameters, assign those hyperparameters to the model, and fit the finalized model on the training set. Previously in tidymodels, this has felt like: # choose a method to define "best" and extract the resulting parameters best_param <- select_best(tuning_res, "rmse") # assign those parameters to model knn_model_final <- finalize_model(knn_model_spec, best_param) # fit the finalized model to the training set knn_fit <- fit(knn_model_final, mpg ~ ., mtcars) Voilà! `knn_fit` is a properly resampled model that is ready to [`predict()`](https://rdrr.io/r/stats/predict.html) on new data: predict(knn_fit, mtcars[1, ]) #> # A tibble: 1 × 1 #> .pred #> #> 1 22.0 The newest release of tune introduced a shorthand interface for going from tuning results to final fit called [`fit_best()`](https://tune.tidymodels.org/reference/fit_best.html) . The function wraps each of those three functions with sensible defaults to abbreviate the process described above. knn_fit_2 <- fit_best(tuning_res) predict(knn_fit_2, mtcars[1, ]) #> # A tibble: 1 × 1 #> .pred #> #> 1 22.0 This function is closely related to the [`last_fit()`](https://tune.tidymodels.org/reference/last_fit.html) function. They both give you access to a workflow fitted on the training data but are situated somewhat differently in the modeling workflow. [`fit_best()`](https://tune.tidymodels.org/reference/fit_best.html) picks up after a tuning function like [`tune_grid()`](https://tune.tidymodels.org/reference/tune_grid.html) to take you from tuning results to fitted workflow, ready for you to predict and assess further. [`last_fit()`](https://tune.tidymodels.org/reference/last_fit.html) assumes you have made your choice of hyperparameters and finalized your workflow to then take you from finalized workflow to fitted workflow and further to performance assessment on the test data. While [`fit_best()`](https://tune.tidymodels.org/reference/fit_best.html) gives a fitted workflow, [`last_fit()`](https://tune.tidymodels.org/reference/last_fit.html) gives you the performance results. If you want the fitted workflow, you can extract it from the result of [`last_fit()`](https://tune.tidymodels.org/reference/last_fit.html) via [`extract_workflow()`](https://hardhat.tidymodels.org/reference/hardhat-extract.html) . The newest release of the workflowsets package also includes a [`fit_best()`](https://tune.tidymodels.org/reference/fit_best.html) method for workflow set objects. Given a set of tuning results, that method will sift through all of the possible models to find and fit the optimal model configuration. Interactive issue logging[](#interactive-issue-logging) -------------------------------------------------------- Imagine, in the previous example, we made some subtle error in specifying the tuning process. For example, passing a function to `extract` elements of the proposed workflows that injects some warnings and errors into the tuning process: raise_concerns <- function(x) { warning("Ummm, wait. :o") stop("Eep! Nooo!") } tuning_res <- tune_grid( knn_model_spec, mpg ~ ., bootstraps(mtcars, 5), control = control_grid(extract = raise_concerns) ) Warnings and errors can come up in all sorts of places while tuning hyperparameters. Often, with obvious issues, we can raise errors early on and halt the tuning process, but with more subtle concerns, we don’t want to be too restrictive; it’s sometimes better to defer to the underlying modeling packages to decide what’s a dire issue versus something that can be worked around. In the past, we’ve raised warnings and issues as they occur, printing context on the issue to the console before logging the issue in the tuning result. In the above example, this would look like: ! Bootstrap1: preprocessor 1/1, model 1/1 (extracts): Ummm, wait. :o x Bootstrap1: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): Eep! Nooo! ! Bootstrap2: preprocessor 1/1, model 1/1 (extracts): Ummm, wait. :o x Bootstrap2: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): Eep! Nooo! ! Bootstrap3: preprocessor 1/1, model 1/1 (extracts): Ummm, wait. :o x Bootstrap3: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): Eep! Nooo! ! Bootstrap4: preprocessor 1/1, model 1/1 (extracts): Ummm, wait. :o x Bootstrap4: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): Eep! Nooo! ! Bootstrap5: preprocessor 1/1, model 1/1 (extracts): Ummm, wait. :o x Bootstrap5: preprocessor 1/1, model 1/1 (extracts): Error in extractor(object): Eep! Nooo! The above messages are super descriptive about where issues occur—they note in which resample, from which proposed modeling workflow, and in which part of the fitting process the issues occurred in. At the same time, they are quite repetitive; if there’s an issue during hyperparameter tuning, it probably occurs in every resample, always in the same place. If, instead, we were evaluating this model against 1,000 resamples, or there were more than just two issues, this output could get very overwhelming very quickly. The new releases of our tuning packages include tools to determine which tuning issues are unique, and for each unique issue, only print out the message once while maintaining a dynamic count of how many times the issue occurred. With the new tune release, the same output would look like: #> → A | warning: Ummm, wait. :o #> → B | error: Eep! Nooo! #> There were issues with some computations A: x5 B: x5 This interface is hopefully less overwhelming for users. When the messages attached to these issues aren’t enough to debug the issue, the complete set of information about the issues lives inside of the tuning result object, and can be retrieved with `collect_notes(tuning_res)`. To turn off the interactive logging, set the `verbose` control option to `TRUE`. Speedups[](#speedups) ---------------------- Each of these three releases, as well as releases of core tidymodels packages they depend on like parsnip, recipes, and hardhat, include a plethora of changes meant to optimize computational performance. Especially for modeling practitioners who work with many resamples and/or small data sets, our modeling workflows will feel a whole lot snappier: ![A ggplot2 line graph plotting relative change in time to evaluate model fits with the tidymodels packages. Fits on datasets with 100 training rows are 2 to 3 times faster, while fits on datasets with 100,000 or more rows take about the same amount of time as they used to.](https://simonpcouch.com/blog/speedups-2023/index_files/figure-html/unnamed-chunk-10-1.png) With 100-row training data sets, the time to resample models with tune and friends has been at least halved. These releases are the first iteration of a set of changes to reduce the evaluation time of tidymodels code, and users can expect further optimizations in coming releases! See [this post on my blog](https://www.simonpcouch.com/blog/speedups-2023/) for more information about those speedups. Bonus points[](#bonus-points) ------------------------------ Although they’re smaller in scope, we wanted to highlight two additional developments in tuning hyperparameters with tidymodels. ### Workflow set support for tidyclust[](#workflow-set-support-for-tidyclust) The recent tidymodels package [tidyclust](github.com/tidymodels/tidyclust) introduced support for fitting and tuning clustering models in tidymodels. That package’s function [`tune_cluster()`](https://tidyclust.tidymodels.org/reference/tune_cluster.html) is now an option for tuning in [`workflow_map()`](https://workflowsets.tidymodels.org/reference/workflow_map.html) , meaning that users can fit sets of clustering models and preprocessors using workflow sets. These changes further integrate the tidyclust package into tidymodels framework. ### Refined retrieval of intermediate results[](#refined-retrieval-of-intermediate-results) The `.Last.tune.result` helper stores the most recent tuning result in the object `.Last.tune.result` as a fail-safe in cases of interrupted tuning, uncaught tuning errors, and simply forgetting to assign tuning results to an object. # be a silly goose and forget to assign results tune_grid( knn_model_spec, mpg ~ ., bootstraps(mtcars, 5), control = control_grid(save_workflow = TRUE) ) #> # Tuning results #> # Bootstrap sampling #> # A tibble: 5 × 4 #> splits id .metrics .notes #> #> 1 Bootstrap1 #> 2 Bootstrap2 #> 3 Bootstrap3 #> 4 Bootstrap4 #> 5 Bootstrap5 # all is not lost! .Last.tune.result #> # Tuning results #> # Bootstrap sampling #> # A tibble: 5 × 4 #> splits id .metrics .notes #> #> 1 Bootstrap1 #> 2 Bootstrap2 #> 3 Bootstrap3 #> 4 Bootstrap4 #> 5 Bootstrap5 # assign to object after the fact res <- .Last.tune.result These three releases introduce support for the `.Last.tune.result` object in more settings and refine support in existing implementations. Acknowledgements[](#acknowledgements) -------------------------------------- Thanks to [@walrossker](https://github.com/walrossker) , [@Freestyleyang](https://github.com/Freestyleyang) , and [@Jeffrothschild](https://github.com/Jeffrothschild) for their contributions to these packages since their last releases. Happy modeling, y’all! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # vctrs [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Vctrs [![](/blog/2023/04/performant-packages/thumbnail-sq.jpg)](/blog/2023/04/performant-packages/) [Writing performant code with tidy tools](/blog/2023/04/performant-packages/) [programming](/categories/programming) Simon Couch When performance becomes an issue for code using tidy interfaces, switching to the backend tools used by tidy developers can offer substantial speedups. [Read more ...](/blog/2023/04/performant-packages/) 2023/04/18 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dials [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Dials [![](/blog/2023/04/tidymodels-2023-q1/thumbnail-sq.jpg)](/blog/2023/04/tidymodels-2023-q1/) [Q1 2023 tidymodels digest](/blog/2023/04/tidymodels-2023-q1/) [roundup](/categories/roundup) Emil Hvitfeldt The tidymodels team has been busy working on all sorts of new features across the ecosystem. [Read more ...](/blog/2023/04/tidymodels-2023-q1/) 2023/04/28 [![](/blog/2022/04/tidymodels-2022-q1/thumbnail-sq.jpg)](/blog/2022/04/tidymodels-2022-q1/) [Q1 2022 tidymodels digest](/blog/2022/04/tidymodels-2022-q1/) [roundup](/categories/roundup) Julia Silge There were 21 releases of tidymodels packages during Q1 of this year, and this post shares some highlights! [Read more ...](/blog/2022/04/tidymodels-2022-q1/) 2022/04/01 [![](/blog/2021/09/tidymodels-2021-q3/thumbnail-sq.jpg)](/blog/2021/09/tidymodels-2021-q3/) [Q3 2021 tidymodels roundup](/blog/2021/09/tidymodels-2021-q3/) [roundup](/categories/roundup) Julia Silge Use new tuning parameters, new recipe steps, and a new example dataset! [Read more ...](/blog/2021/09/tidymodels-2021-q3/) 2021/09/28 [![](/blog/2019/10/dials-0-0-3/dials-0-0-3-sq.jpg)](/blog/2019/10/dials-0-0-3/) [dials 0.0.3](/blog/2019/10/dials-0-0-3/) [package](/categories/package) Max Kuhn A major update to the tuning parameter package. [Read more ...](/blog/2019/10/dials-0-0-3/) 2019/10/01 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr 1.1.1 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dplyr 1.1.1 =========== ![](/blog/2023/03/dplyr-1-1-1/thumbnail-wd.jpg) Photo by [Jon Tyson](https://unsplash.com/photos/jy8z4NBIYSQ)   2023/03/22   [dplyr](/tags/dplyr/) , [dplyr-1-1-0](/tags/dplyr-1-1-0/)   Davis Vaughan We’re stoked to announce the release of [dplyr 1.1.1](https://dplyr.tidyverse.org/) . We don’t typically blog about patch releases, because they generally only fix bugs without significantly changing behavior, but this one includes two important updates: * Addressing various performance regressions * Refining the `multiple` match warning thrown by dplyr’s joins You can see a full list of changes in the [release notes](https://dplyr.tidyverse.org/news/index.html) . To see the other blog posts in the dplyr 1.1.0 series, head [here](https://www.tidyverse.org/tags/dplyr-1-1-0/) . You can install dplyr 1.1.1 from CRAN with: install.packages("dplyr") library(dplyr) Performance regressions[](#performance-regressions) ---------------------------------------------------- In the [1.1.0 post on vctrs](/blog/2023/02/dplyr-1-1-0-vctrs) , we discussed that we’ve rewritten all of dplyr’s vector functions on top of [vctrs](https://vctrs.r-lib.org/) for improved versatility. Unfortunately, we accidentally made two sets of functions much slower, especially when used on a data frame with many groups: * [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) and [`if_else()`](https://dplyr.tidyverse.org/reference/if_else.html) * [`nth()`](https://dplyr.tidyverse.org/reference/nth.html) , [`first()`](https://dplyr.tidyverse.org/reference/nth.html) , and [`last()`](https://dplyr.tidyverse.org/reference/nth.html) These performance issues have been addressed, and should be back to 1.0.10 level of performance. [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) is still _slightly_ slower than 1.0.10, but it isn’t likely to be very noticeable, and we already have plans to improve this further in a future release. Revisiting multiple matches[](#revisiting-multiple-matches) ------------------------------------------------------------ In the [1.1.0 post on joins](/blog/2023/01/dplyr-1-1-0-joins) , we discussed the new `multiple` argument that was added to [`left_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) and friends, which had a built in safety check that warned when you performed a join where a row from `x` matched more than one row from `y`. The TLDR of the discussion below is that we’ve realized that this warning was being thrown in too many cases, so we’ve adjusted it in such a way that it now only catches the most dangerous type of join (a many-to-many join), meaning that you should see the warning _much_ less often. As a reminder, `multiple` determines what happens when a row from `x` matches more than one row from `y`. You can choose to return `"all"` of the matches, the `"first"` or `"last"` match, or `"any"` of the matches if you are just interested in detecting if there is at least one. `multiple` defaulted to a behavior similar to `"all"`, with the added side effect of throwing a warning if multiple matches were actually detected, like this: student <- tibble( student_id = c(1, 2, 3), transfer = c(FALSE, TRUE, TRUE), initial_term = c("fall 2018", "fall 2020", "fall 2020") ) term <- tibble( student_id = c(1, 1, 2, 3, 3, 3), term = c("fall 2018", "spring 2019", "fall 2020", "fall 2020", "spring 2021", "fall 2021"), course_load = c(12, 15, 10, 14, 15, 12) ) # Information about students attending a university. # One row per (student_id). student #> # A tibble: 3 × 3 #> student_id transfer initial_term #> #> 1 1 FALSE fall 2018 #> 2 2 TRUE fall 2020 #> 3 3 TRUE fall 2020 # Term specific information about each student. # One row per (student_id, term) combination. term #> # A tibble: 6 × 3 #> student_id term course_load #> #> 1 1 fall 2018 12 #> 2 1 spring 2019 15 #> 3 2 fall 2020 10 #> 4 3 fall 2020 14 #> 5 3 spring 2021 15 #> 6 3 fall 2021 12 student |> left_join(term, join_by(student_id)) #> Warning in left_join(student, term, join_by(student_id)): Each row in `x` is expected to match at most 1 row in `y`. #> i Row 1 of `x` matches multiple rows. #> i If multiple matches are expected, set `multiple = "all"` to silence this warning. #> # A tibble: 6 × 5 #> student_id transfer initial_term term course_load #> #> 1 1 FALSE fall 2018 fall 2018 12 #> 2 1 FALSE fall 2018 spring 2019 15 #> 3 2 TRUE fall 2020 fall 2020 10 #> 4 3 TRUE fall 2020 fall 2020 14 #> 5 3 TRUE fall 2020 spring 2021 15 #> 6 3 TRUE fall 2020 fall 2021 12 To silence this warning, we encouraged you to set `multiple = "all"` to be explicit about the fact that you expected a row from `x` to match multiple rows in `y`. The original motivation for this behavior comes from a two-part hypothesis of ours: * Users are often surprised when a join returns more rows than the left-hand table started with (in the above example, `student` has 3 rows but the join result has 6). * It is dangerous to allow joins that can result in a Cartesian explosion of the number of rows (i.e. `nrow(x) * nrow(y)`). This hypothesis led us to automatically warn on two types of join relationships, one-to-many joins and many-to-many joins. If you aren’t familiar with these terms, here is a quick rundown of the 4 types of join relationships (often discussed in a SQL context), which provide constraints on the number of allowed matches: * one-to-one: * A row from `x` can match at most 1 row from `y`. * A row from `y` can match at most 1 row from `x`. * one-to-many: * A row from `x` can match any number of rows in `y`. * A row from `y` can match at most 1 row from `x`. * many-to-one: * A row from `x` can match at most 1 row from `y`. * A row from `y` can match any number of rows in `x`. * many-to-many: * A row from `x` can match any number of rows in `y`. * A row from `y` can match any number of rows in `x`. After gathering some valuable [user feedback](https://github.com/tidyverse/dplyr/issues/6717) and conducting an [in depth analysis](https://github.com/tidyverse/dplyr/issues/6731) of these join relationships, we’ve determined that the only relationship style actually worth warning on is many-to-many, because that is the one that can result in a Cartesian explosion of rows. In retrospect, the one-to-many relationship is actually quite common, and is symmetrical with many-to-one, which we weren’t warning on. You could actually exploit this fact by switching the above join around, which would silence the warning: term |> left_join(student, join_by(student_id)) #> # A tibble: 6 × 5 #> student_id term course_load transfer initial_term #> #> 1 1 fall 2018 12 FALSE fall 2018 #> 2 1 spring 2019 15 FALSE fall 2018 #> 3 2 fall 2020 10 TRUE fall 2020 #> 4 3 fall 2020 14 TRUE fall 2020 #> 5 3 spring 2021 15 TRUE fall 2020 #> 6 3 fall 2021 12 TRUE fall 2020 We still believe that new users are often surprised when a join returns more rows than they originally started with, but the many-to-one case of this is rarely a problem in practice. So, as of dplyr 1.1.1, we no longer warn on one-to-many relationships, which should drastically reduce the amount of warnings that you see. ### Many-to-many relationships[](#many-to-many-relationships) A many-to-many relationship is much harder to construct (which is good). In fact, a database system won’t even let you create one of these “relationships” between two tables directly, instead requiring you to create a third bridge table that turns the many-to-many relationship into two one-to-many relationships. We can “accidentally” create one of these in R though: course <- tibble( student_id = c(1, 1, 1, 2, 2, 3, 3, 3, 3), instructor_id = c(1, 2, 3, 1, 2, 1, 2, 3, 4), course = c(101, 110, 123, 110, 101, 110, 115, 110, 101), term = c( "fall 2018", "fall 2018", "spring 2019", "fall 2020", "fall 2020", "fall 2020", "fall 2020", "spring 2021", "fall 2021" ), grade = c("A", "B", "A", "B", "C", "A", "C", "D", "B") ) # Information about the courses each student took per semester. # One row per (student_id, course, term) combination. course #> # A tibble: 9 × 5 #> student_id instructor_id course term grade #> #> 1 1 1 101 fall 2018 A #> 2 1 2 110 fall 2018 B #> 3 1 3 123 spring 2019 A #> 4 2 1 110 fall 2020 B #> 5 2 2 101 fall 2020 C #> 6 3 1 110 fall 2020 A #> 7 3 2 115 fall 2020 C #> 8 3 3 110 spring 2021 D #> 9 3 4 101 fall 2021 B # Forgetting to join by both `student_id` and `term`! term |> left_join(course, by = join_by(student_id)) #> Warning in left_join(term, course, by = join_by(student_id)): Detected an unexpected many-to-many relationship between `x` and `y`. #> ℹ Row 1 of `x` matches multiple rows in `y`. #> ℹ Row 1 of `y` matches multiple rows in `x`. #> ℹ If a many-to-many relationship is expected, set `relationship = #> "many-to-many"` to silence this warning. #> # A tibble: 20 × 7 #> student_id term.x course_load instructor_id course term.y grade #> #> 1 1 fall 2018 12 1 101 fall 2018 A #> 2 1 fall 2018 12 2 110 fall 2018 B #> 3 1 fall 2018 12 3 123 spring 2019 A #> 4 1 spring 2019 15 1 101 fall 2018 A #> 5 1 spring 2019 15 2 110 fall 2018 B #> 6 1 spring 2019 15 3 123 spring 2019 A #> 7 2 fall 2020 10 1 110 fall 2020 B #> 8 2 fall 2020 10 2 101 fall 2020 C #> 9 3 fall 2020 14 1 110 fall 2020 A #> 10 3 fall 2020 14 2 115 fall 2020 C #> 11 3 fall 2020 14 3 110 spring 2021 D #> 12 3 fall 2020 14 4 101 fall 2021 B #> 13 3 spring 2021 15 1 110 fall 2020 A #> 14 3 spring 2021 15 2 115 fall 2020 C #> 15 3 spring 2021 15 3 110 spring 2021 D #> 16 3 spring 2021 15 4 101 fall 2021 B #> 17 3 fall 2021 12 1 110 fall 2020 A #> 18 3 fall 2021 12 2 115 fall 2020 C #> 19 3 fall 2021 12 3 110 spring 2021 D #> 20 3 fall 2021 12 4 101 fall 2021 B In the example above, we’ve forgotten to include the `term` column when joining these two tables together, which accidentally results in a small explosion of rows (we end up with 20 rows, more than in either original input, but not quite the maximum possible amount, which is a whopping 54 rows!). Luckily, dplyr warns us that at least one row in each table matches more than one row in the opposite table - a sign that something isn’t right. At this point we can do one of two things: * Look into the new `relationship` argument that the warning mentions (we’ll discuss this below) * Look at our join to see if we made a mistake Of course, in this case we’ve messed up, and adding `term` into the by expression results in the correct (and silent) join: term |> left_join(course, by = join_by(student_id, term)) #> # A tibble: 9 × 6 #> student_id term course_load instructor_id course grade #> #> 1 1 fall 2018 12 1 101 A #> 2 1 fall 2018 12 2 110 B #> 3 1 spring 2019 15 3 123 A #> 4 2 fall 2020 10 1 110 B #> 5 2 fall 2020 10 2 101 C #> 6 3 fall 2020 14 1 110 A #> 7 3 fall 2020 14 2 115 C #> 8 3 spring 2021 15 3 110 D #> 9 3 fall 2021 12 4 101 B ### Join `relationship`s[](#join-relationships) To adjust the joins to only warn on many-to-many relationships, we’ve done two things: * `multiple` now defaults to `"all"`, and is now focused solely on limiting the matches returned if multiple are detected, rather than also optionally warning/erroring. * We’ve added a new `relationship` argument. The `relationship` argument allows you to explicitly specify the expected join relationship between the keys of `x` and `y` using the exact options we listed above: `"one-to-one"`, `"one-to-many"`, `"many-to-one"`, and `"many-to-many"`. If the constraints of the relationship you choose are violated, an error is thrown. For example, we could use this to require that the `student` + `term` join contains a one-to-many relationship between the two tables: student |> left_join(term, join_by(student_id), relationship = "one-to-many") #> # A tibble: 6 × 5 #> student_id transfer initial_term term course_load #> #> 1 1 FALSE fall 2018 fall 2018 12 #> 2 1 FALSE fall 2018 spring 2019 15 #> 3 2 TRUE fall 2020 fall 2020 10 #> 4 3 TRUE fall 2020 fall 2020 14 #> 5 3 TRUE fall 2020 spring 2021 15 #> 6 3 TRUE fall 2020 fall 2021 12 Let’s violate this by adding a duplicate row in `student`: student_bad <- student |> tibble::add_row( student_id = 1, transfer = FALSE, initial_term = "fall 2019", .after = 1 ) student_bad #> # A tibble: 4 × 3 #> student_id transfer initial_term #> #> 1 1 FALSE fall 2018 #> 2 1 FALSE fall 2019 #> 3 2 TRUE fall 2020 #> 4 3 TRUE fall 2020 student_bad |> left_join(term, join_by(student_id), relationship = "one-to-many") #> Error in `left_join()`: #> ! Each row in `y` must match at most 1 row in `x`. #> ℹ Row 1 of `y` matches multiple rows in `x`. The default value of `relationship` doesn’t add any constraints, but for equality joins it will check to see if a many-to-many relationship exists, and will warn if one occurs (like with the `term` + `course` join from above). As mentioned before, this is quite hard to do, and often means you have a mistake in your join call or in the data itself. If you really do want to perform a join with this kind of relationship, to silence the warning you can explicitly specify `relationship = "many-to-many"`. One last thing to note is that `relationship` doesn’t handle the case of an _unmatched_ row. For that, you should use the `unmatched` argument that was also added in 1.1.0. The combination of `relationship` and `unmatched` provides a complete set of tools for adding production level quality control checks to your joins. Acknowledgements[](#acknowledgements) -------------------------------------- The examples used in this blog post were adapted from [@eipi10](https://github.com/eipi10) in [this issue](https://github.com/tidyverse/dplyr/issues/6717) . We’d like to thank all 66 contributors who help in someway, whether it was filing issues or contributing code and documentation: [@alexhallam](https://github.com/alexhallam) , [@ammar-gla](https://github.com/ammar-gla) , [@arnaudgallou](https://github.com/arnaudgallou) , [@ArthurAndrews](https://github.com/ArthurAndrews) , [@AuburnEagle-578](https://github.com/AuburnEagle-578) , [@batpigandme](https://github.com/batpigandme) , [@billdenney](https://github.com/billdenney) , [@Bisaloo](https://github.com/Bisaloo) , [@bitplane](https://github.com/bitplane) , [@chrarnold](https://github.com/chrarnold) , [@D5n9sMatrix](https://github.com/D5n9sMatrix) , [@daattali](https://github.com/daattali) , [@DanChaltiel](https://github.com/DanChaltiel) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dieghernan](https://github.com/dieghernan) , [@dkutner](https://github.com/dkutner) , [@eipi10](https://github.com/eipi10) , [@eitsupi](https://github.com/eitsupi) , [@emilBeBri](https://github.com/emilBeBri) , [@fawda123](https://github.com/fawda123) , [@fedassembly](https://github.com/fedassembly) , [@fkohrt](https://github.com/fkohrt) , [@gavinsimpson](https://github.com/gavinsimpson) , [@geogale](https://github.com/geogale) , [@ggrothendieck](https://github.com/ggrothendieck) , [@hadley](https://github.com/hadley) , [@hope-data-science](https://github.com/hope-data-science) , [@jaganmn](https://github.com/jaganmn) , [@jakub-jedrusiak](https://github.com/jakub-jedrusiak) , [@JorisChau](https://github.com/JorisChau) , [@krlmlr](https://github.com/krlmlr) , [@krprasangdas](https://github.com/krprasangdas) , [@larry77](https://github.com/larry77) , [@lionel-](https://github.com/lionel-) , [@lschneiderbauer](https://github.com/lschneiderbauer) , [@LukasWallrich](https://github.com/LukasWallrich) , [@maellecoursonnais](https://github.com/maellecoursonnais) , [@manhnguyen48](https://github.com/manhnguyen48) , [@mattansb](https://github.com/mattansb) , [@mgirlich](https://github.com/mgirlich) , [@mhaynam](https://github.com/mhaynam) , [@MichaelChirico](https://github.com/MichaelChirico) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@mkoohafkan](https://github.com/mkoohafkan) , [@moodymudskipper](https://github.com/moodymudskipper) , [@Moohan](https://github.com/Moohan) , [@msgoussi](https://github.com/msgoussi) , [@multimeric](https://github.com/multimeric) , [@osheen1](https://github.com/osheen1) , [@Pozdniakov](https://github.com/Pozdniakov) , [@psychelzh](https://github.com/psychelzh) , [@pur80a](https://github.com/pur80a) , [@robayo](https://github.com/robayo) , [@rszulkin](https://github.com/rszulkin) , [@salim-b](https://github.com/salim-b) , [@sda030](https://github.com/sda030) , [@sfirke](https://github.com/sfirke) , [@shannonpileggi](https://github.com/shannonpileggi) , [@stephLH](https://github.com/stephLH) , [@szabgab](https://github.com/szabgab) , [@tjebo](https://github.com/tjebo) , [@Torvaney](https://github.com/Torvaney) , [@twest820](https://github.com/twest820) , [@vanillajonathan](https://github.com/vanillajonathan) , [@warnes](https://github.com/warnes) , and [@zknitter](https://github.com/zknitter) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dtplyr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Dtplyr [![](/blog/2023/02/dtplyr-1-3-0/thumbnail-sq.jpg)](/blog/2023/02/dtplyr-1-3-0/) [dtplyr 1.3.0](/blog/2023/02/dtplyr-1-3-0/) [package](/categories/package) Hadley Wickham dtplyr brings initial support for dplyr 1.1.0 features, new translations, and a breaking change. [Read more ...](/blog/2023/02/dtplyr-1-3-0/) 2023/02/24 [![](/blog/2019/11/dtplyr-1-0-0/thumb-sq.jpg)](/blog/2019/11/dtplyr-1-0-0/) [dtplyr 1.0.0](/blog/2019/11/dtplyr-1-0-0/) [package](/categories/package) Hadley Wickham A total rewrite of dtplyr is now available on CRAN; it performs computation lazily (like dbplyr), making it much more performant. [Read more ...](/blog/2019/11/dtplyr-1-0-0/) 2019/11/12 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # New CRAN requirements for packages with C and C++ [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) New CRAN requirements for packages with C and C++ ================================================= ![](/blog/2023/03/cran-checks-compiled-code/thumbnail-wd.jpg) Photo by [Quino Al](https://unsplash.com/photos/dhGFLj3rI0Q)   2023/03/30   Andy Teucher The R package landscape is dynamic, with changes in infrastructure common, especially when CRAN makes changes to their policies and requirements. This is particularly true for packages that include low-level compiled code, requiring developers to be nimble in responding to these changes. The tidyverse team at Posit is in the unique situation where we have a concentration of developers working full-time on creating and maintaining open source packages. This internal community provides the opportunity to collaborate to develop shared practices and discover solutions to problems that arise. When we can, we like to share what we’ve learned so other developers can benefit. There have been a few recent changes at CRAN for packages containing C and C++ code that developers have had to adapt to, and we would like to share some of our learning: NOTE regarding `SystemRequirements: C++11`[](#note-regarding-systemrequirements-c11) ------------------------------------------------------------------------------------- Many package authors might have noticed a new NOTE on R-devel when submitting a package to CRAN containing C++ code: * checking C++ specification ... NOTE Specified C++11: please drop specification unless essential This NOTE is now appearing during `R CMD check` on R-devel for packages where the DESCRIPTION file has the following: SystemRequirements: C++11 Packages that use C++11 would also usually have set `CXX_STD=CXX11` in the `src/Makevars` and `src/Makevars.win` files (and `src/Makevars.ucrt`, if present). These specifications tell R to use the C++11 standard when compiling the code. To understand the NOTE, a bit of history will be helpful (thanks to Winston Chang for [writing this up](https://gist.github.com/wch/849ca79c9416795d99c48cc06a44ca1e) ): * In R 3.5 and below, on systems with an old compiler, R would default to using the C++98 standard when compiling the code. If a package needed a C++11 compiler, the DESCRIPTION file needed to have `SystemRequirements: C++11`, and the various `src/Makevars*` files needed to set `CXX_STD=CXX11`. * In R 3.6.2, R began defaulting to compiling packages with the C++11 standard, as long as the compiler supported C++11 (which was true on most systems). * In R 4.0, C++11 became the minimum supported compiler, so `SystemRequirements: C++11` was no longer necessary. * In (the forthcoming) R 4.3, the [default C++ standard is C++17](https://developer.r-project.org/blosxom.cgi/R-devel/NEWS/2023/01/27#n2023-01-27) where available. `R CMD check` now [raises a NOTE](https://developer.r-project.org/blosxom.cgi/R-devel/NEWS/2023/01/31) if anything older than the default is specified in `SystemRequirements:` or `CXX_STD` in the various `src/Makevars*` files. This NOTE will block submission to CRAN — if the standard you specify is necessary for your package you will likely need to explain why. ### How to fix it[](#how-to-fix-it) 1. Edit the DESCRIPTION file and remove `SystemRequirements: C++11`. 2. Edit `src/Makevars`, `src/Makevars.win`, and `src/Makevars.ucrt` and remove `CXX_STD=CXX11`. After making these changes, the package should install without trouble on R 3.6 and above. It may not build on R <= 3.5 on systems with very old compilers, though it is likely that the vast majority of users will have a newer version of R and/or have recent enough compilers. If you want to be confident that your package will be installable on R 3.5 and below with old compilers, there are several options; we offer two of the simplest approaches here: * You can use a configure script at the top level of the package, and have it add `CXX_STD=CXX11` for R 3.5 and below. An example (unmerged) [pull request to the readxl](https://github.com/tidyverse/readxl/pull/722/files) package demonstrates this approach. You will also need to add `Biarch: true` in your DESCRIPTION file. This appears to be the approach preferred by CRAN. * For users with R <= 3.5 on a system with an older compiler, package authors can instruct users to edit their `~/.R/Makevars` file to include this line: `CXX_STD=CXX11`. The tidyverse has a [policy of supporting four previous versions](https://www.tidyverse.org/blog/2019/04/r-version-support/) of R. Currently that includes R 3.5, but with the upcoming release of R 4.3 (which should be this Spring some time) the minimum version we will support is R 3.6. As we won’t be supporting R 3.5 in the near future, you should not feel pressured to either. WARNING regarding the use of `sprintf()` in C/C++[](#warning-regarding-the-use-of-codesprintfcode-in-cc) --------------------------------------------------------------------------------------------------------- Another recent change in CRAN checks on R-devel that authors might encounter is the disallowing of the use of the C functions `sprintf()` and `vsprintf()`. `R CMD check` on R-devel may throw warnings that look something like this: checking compiled code ... WARNING File 'fs/libs/fs.so': Found 'sprintf', possibly from 'sprintf' (C) Object: 'file.o' Compiled code should not call entry points which might terminate R nor write to stdout/stderr instead of to the console, nor use Fortran I/O nor system RNGs nor [v]sprintf. See 'Writing portable packages' in the 'Writing R Extensions' manual. According to the [NEWS for R-devel](https://developer.r-project.org/blosxom.cgi/R-devel/NEWS/2022/12/24#n2022-12-24) (which will be R 4.3): > The use of sprintf and vsprintf from C/C++ has been deprecated in macOS 13 and is a known security risk. `R CMD check` now reports (on all platforms) if their use is found in compiled code: replace by snprintf or vsnprintf respectively. These are considered to be a security risk because they potentially allow [buffer overflows](https://en.wikipedia.org/wiki/Buffer_overflow) that write more bytes than are available in the output buffer. This is a risk if the text that is being passed to `sprintf()` comes from an uncontrolled source. Here is a very simple example: library(cpp11) cpp_function(' int say_height(int height) { // "My height is xxx cm" is 19 characters but we need // to add one for the null-terminator char out[19 + 1]; int n; n = sprintf(out, "My height is %i cm", height); Rprintf(out); return n; } ' ) say_height(182) #> My height is 182 cm #> [1] 19 say_height(1824) # This will abort due to buffer overflow ### How to fix it[](#how-to-fix-it-1) In most cases, this should be a simple fix: replace `sprintf()` with `snprintf()` and `vsprintf()` with `vsnprintf()`. These `n` variants take a second parameter `size`, that specifies the maximum number of bytes to be written, _including the automatically appended null-terminator_. If the output is a static buffer, you can use `sizeof()`: cpp_function(' int say_height_safely(int height) { // "My height is xxx cm\\n" is 20 characters but we need // to add one for the null-terminator char out[20 + 1]; int n; n = snprintf(out, sizeof(out), "My height is %i cm\\n", height); Rprintf(out); return n; } ') say_height_safely(182) #> My height is 182 cm #> [1] 20 say_height_safely(1824567) #> My height is 1824567 #> [1] 24 Notice that the return value of `sprintf()` and `snprintf()` are slightly different. `sprintf()` returns the total number of characters written (excluding the null-terminator), while `snprintf()` returns the length of the formatted string, whether or not it has been truncated to match `size`. It is a bit trickier if the destination is not a static buffer, so you’ll have to determine the maximum `size` by carefully thinking about the code. WARNING regarding the use of strict prototypes in C[](#warning-regarding-the-use-of-strict-prototypes-in-c) ------------------------------------------------------------------------------------------------------------ Many maintainers with packages containing C code have also been getting hit with this warning: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] This usually comes from C function declarations that look like this, with no arguments specified (which is very common): int myfun() { ... }; This new warning is because CRAN is now running checks on R-devel with the `-Wstrict-prototypes` compiler flag set. In R we define functions that take no arguments with `myfun <- function() {...}` all the time. In C, with this flag set, the fact that a function takes no arguments must be explicitly stated (i.e., the arguments list cannot be empty). In the upcoming C23 standard, empty function signatures will be considered valid and not ambiguous, however at this point it is likely to be the reason you encounter this warning from CRAN. ### How to fix it[](#how-to-fix-it-2) This can be fixed by placing the `void` keyword in the previously empty argument list: int myfun(void) { ... }; Here is an example where the authors of [Cubist](https://topepo.github.io/Cubist/) applied the [necessary patches](https://github.com/topepo/Cubist/pull/46) , and [another one in rlang](https://github.com/r-lib/rlang/pull/1508) . ### Vendored code[](#vendored-code) Function declarations without a prototype are very common, and unfortunately are thus likely to appear in libraries that you include in your package. This may require you to patch that code in your package. The [readxl](https://readxl.tidyverse.org) package includes the [libxls C library](https://github.com/libxls/libxls) , which was patched [in readxl here](https://github.com/tidyverse/readxl/commit/afdc9b90cfc2bb1e1c5490c7ba3af5ecfc4a7876) to deal with this issue. The ideal solution in cases like this would be to submit patches to the upstream libraries so you don’t have to deal with the ongoing maintenance of your local patches, but that is not always possible. Generally, you can explain this problem when submitting your package, and as long as you’ve have notified the upstream maintainer, CRAN should accept your updated package. ### Unspecified types in function signature[](#unspecified-types-in-function-signature) The `-Wstrict-prototypes` compiler flag will also catch deprecated function definitions where the types of the arguments are not declared. This is actually likely the primary purpose for CRAN enabling this flag, as it is ambiguous and much more dangerous than empty function signatures. These take the form: void myfun(x, y) { ... }; where the argument types are not declared. This is solved by declaring the types of the arguments: void myfun(int x, char* y) { ... }; Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyverse 2.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) tidyverse 2.0.0 =============== ![](/blog/2023/03/tidyverse-2-0-0/thumbnail-wd.jpg) Photo by [Graham Holtshausen](https://unsplash.com/photos/fUnfEz3VLv4)   2023/03/08   [tidyverse](/tags/tidyverse/)   Hadley Wickham We’re tickled pink to announce the release of [tidyverse](http://tidyverse.tidyverse.org/) 2.0.0. The tidyverse is a set of packages that work in harmony because they share common data representations and API design. The tidyverse package is a “meta” package designed to make it easy to install and load core packages from the tidyverse in a single command. This is great for teaching and interactive use, but for package-development purposes we recommend that authors import only the specific packages that they use. For a complete list of changes, please see the release notes. You can install it from CRAN with: install.packages("tidyverse") There’s only really one big change in tidyverse 2.0.0: lubridate is now a core member of the tidyverse! This means it’s attached automatically when you load the tidyverse: library(tidyverse) #> ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ── #> ✔ dplyr 1.1.0.9000 ✔ readr 2.1.4 #> ✔ forcats 1.0.0 ✔ stringr 1.5.0 #> ✔ ggplot2 3.4.1 ✔ tibble 3.1.8 #> ✔ lubridate 1.9.2 ✔ tidyr 1.3.0 #> ✔ purrr 1.0.1 #> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ℹ Use the conflicted package to force all conflicts to become errors You’ll notice one other small change to the tidyverse message: we now advertise the [conflicted package](https://conflicted.r-lib.org) . This package has been around for a while, but we wanted to promote it a bit more heavily because it’s so useful. conflicted provides an alternative conflict resolution strategy, when multiple packages export a function of the same name. R’s default conflict resolution system gives precedence to the most recently loaded package. This can make it hard to detect conflicts, particularly when they’re introduced by an update to an existing package. conflicted takes a different approach, turning conflicts into errors and forcing you to choose which function to use. To use conflicted, all you need to do is load it: library(conflicted) Using any function that’s defined in multiple packages will now throw an error: filter(mtcars, cyl == 8) #> Error: #> ! [conflicted] filter found in 2 packages. #> Either pick the one you want with `::`: #> • dplyr::filter #> • stats::filter #> Or declare a preference with `conflicts_prefer()`: #> • `conflicts_prefer(dplyr::filter)` #> • `conflicts_prefer(stats::filter)` As the error suggests, to resolve the problem you can either namespace individual calls: dplyr::filter(mtcars, am & cyl == 8) #> mpg cyl disp hp drat wt qsec vs am gear carb #> Ford Pantera L 15.8 8 351 264 4.22 3.17 14.5 0 1 5 4 #> Maserati Bora 15.0 8 301 335 3.54 3.57 14.6 0 1 5 8 Or declare a session wide preference: conflicts_prefer(dplyr::filter()) #> [conflicted] Will prefer dplyr::filter over any other package. filter(mtcars, am & cyl == 8) #> mpg cyl disp hp drat wt qsec vs am gear carb #> Ford Pantera L 15.8 8 351 264 4.22 3.17 14.5 0 1 5 4 #> Maserati Bora 15.0 8 301 335 3.54 3.57 14.6 0 1 5 8 The conflicted package is fairly established, but it hasn’t seen a huge amount of use, so if you think of something that would make it better, [please let us know!](https://github.com/r-lib/conflicted/issues) . The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr 1.1.0: The power of vctrs [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dplyr 1.1.0: The power of vctrs =============================== ![](/blog/2023/02/dplyr-1-1-0-vctrs/thumbnail-wd.jpg) Photo by [Armand Khoury](https://unsplash.com/photos/n6vS3xlnsCc)   2023/02/02   [dplyr](/tags/dplyr/) , [dplyr-1-1-0](/tags/dplyr-1-1-0/)   Davis Vaughan Today’s [dplyr 1.1.0](https://dplyr.tidyverse.org/news/index.html#dplyr-110) post is focused on various updates to vector functions, like [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) and [`between()`](https://dplyr.tidyverse.org/reference/between.html) . If you missed our previous posts, you can also see the other [blog posts](https://www.tidyverse.org/tags/dplyr-1-1-0/) in this series. All of dplyr’s vector functions are now backed by [vctrs](https://vctrs.r-lib.org/) , which typically results in better error messages, better performance, and greater versatility. install.packages("dplyr") library(dplyr) `case_when()`[](#case_when) ---------------------------- If you’ve used [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) before, you’ve probably written a statement like this: x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", x < 0 ~ NA ) #> Error: `NA` must be , not . Like me, you’ve probably forgotten that [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) has historically been strict about the types on the right-hand side of the `~`, which means that I needed to use `NA_character_` here instead of `NA`. Luckily, the switch to vctrs means that the above code now “just works”: case_when( x >= 10 ~ "large", x >= 0 ~ "small", x < 0 ~ NA ) #> [1] "small" "large" NA "small" NA NA "small" You’ve probably also written a statement like this: case_when( x >= 10 ~ "large", x >= 0 ~ "small", is.na(x) ~ "missing", TRUE ~ "other" ) #> [1] "small" "large" "other" "small" "other" "missing" "small" In this case, we have a fall-through “default” captured by `TRUE ~`. This has always felt a little awkward and is fairly difficult to explain to new R users. To make this clearer, we’ve added an explicit `.default` argument that we encourage you to use instead: case_when( x >= 10 ~ "large", x >= 0 ~ "small", is.na(x) ~ "missing", .default = "other" ) #> [1] "small" "large" "other" "small" "other" "missing" "small" `.default` will always be processed last, regardless of where you put it in the call to [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) , so we recommend placing it at the very end. We haven’t started any formal deprecation process for `TRUE ~` yet, but now that there is a better solution available we encourage you to switch over. We do plan to deprecate this feature in the future because it involves some slightly problematic recycling rules (but we wouldn’t even begin this process for at least a year). `case_match()`[](#case_match) ------------------------------ Another type of [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) statement you’ve probably written is some kind of value remapping like: x <- c("USA", "Canada", "Wales", "UK", "China", NA, "Mexico", "Russia") case_when( x %in% c("USA", "Canada", "Mexico") ~ "North America", x %in% c("Wales", "UK") ~ "Europe", x %in% "China" ~ "Asia" ) #> [1] "North America" "North America" "Europe" "Europe" #> [5] "Asia" NA "North America" NA Remapping values in this way is so common that SQL gives it its own name - the “simple” case statement. To streamline this further, we’ve taken out some of the repetition involved with `x %in%` by introducing [`case_match()`](https://dplyr.tidyverse.org/reference/case_match.html) , a variant of [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) that allows you to specify one or more _values_ on the left-hand side of the `~`, rather than logical vectors. case_match( x, c("USA", "Canada", "Mexico") ~ "North America", c("France", "UK") ~ "Europe", "China" ~ "Asia" ) #> [1] "North America" "North America" NA "Europe" #> [5] "Asia" NA "North America" NA I think that [`case_match()`](https://dplyr.tidyverse.org/reference/case_match.html) is particularly neat because it can be wrapped into an ad-hoc replacement helper if you just need to collapse or replace a few problematic values in a vector, while leaving everything else unchanged: replace_match <- function(x, ...) { case_match(x, ..., .default = x, .ptype = x) } replace_match( x, "USA" ~ "United States", c("UK", "Wales") ~ "United Kingdom", NA ~ "[Missing]" ) #> [1] "United States" "Canada" "United Kingdom" "United Kingdom" #> [5] "China" "[Missing]" "Mexico" "Russia" `consecutive_id()`[](#consecutive_id) -------------------------------------- At Posit, we have regular company update meetings. Since we are all remote, these meetings are over Zoom. Zoom has a neat feature where it can record the transcript of your call, and it will report who was speaking and what they said. It looks something like this: transcript <- tribble( ~name, ~text, "Hadley", "I'll never learn Python.", "Davis", "But aren't you speaking at PyCon?", "Hadley", "So?", "Hadley", "That doesn't influence my decision.", "Hadley", "I'm not budging!", "Mara", "Typical, Hadley. Stubborn as always.", "Davis", "Fair enough!", "Davis", "Let's move on." ) transcript #> # A tibble: 8 × 2 #> name text #> #> 1 Hadley I'll never learn Python. #> 2 Davis But aren't you speaking at PyCon? #> 3 Hadley So? #> 4 Hadley That doesn't influence my decision. #> 5 Hadley I'm not budging! #> 6 Mara Typical, Hadley. Stubborn as always. #> 7 Davis Fair enough! #> 8 Davis Let's move on. We were working with this data and wanted a way to collapse each continuous thought down to one line. For example, rows 3-5 all contain a single idea from Hadley, so we’d like those to be collapsed into a single line. This isn’t quite as straightforward as a simple group-by-`name` and [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) : transcript |> summarise(text = stringr::str_flatten(text, collapse = " "), .by = name) #> # A tibble: 3 × 2 #> name text #> #> 1 Hadley I'll never learn Python. So? That doesn't influence my decision. I'm n… #> 2 Davis But aren't you speaking at PyCon? Fair enough! Let's move on. #> 3 Mara Typical, Hadley. Stubborn as always. This isn’t quite right because it collapsed the first row where Hadley says “I’ll never learn Python” alongside rows 3-5. We need a way to identify consecutive _runs_ representing when a single person is speaking, which is exactly what [`consecutive_id()`](https://dplyr.tidyverse.org/reference/consecutive_id.html) is for! transcript |> mutate(id = consecutive_id(name)) #> # A tibble: 8 × 3 #> name text id #> #> 1 Hadley I'll never learn Python. 1 #> 2 Davis But aren't you speaking at PyCon? 2 #> 3 Hadley So? 3 #> 4 Hadley That doesn't influence my decision. 3 #> 5 Hadley I'm not budging! 3 #> 6 Mara Typical, Hadley. Stubborn as always. 4 #> 7 Davis Fair enough! 5 #> 8 Davis Let's move on. 5 [`consecutive_id()`](https://dplyr.tidyverse.org/reference/consecutive_id.html) takes one or more columns and generates an integer vector that increments every time a value in one of those columns changes. This gives us something we can group on to correctly flatten our `text`. transcript |> mutate(id = consecutive_id(name)) |> summarise(text = stringr::str_flatten(text, collapse = " "), .by = c(id, name)) #> # A tibble: 5 × 3 #> id name text #> #> 1 1 Hadley I'll never learn Python. #> 2 2 Davis But aren't you speaking at PyCon? #> 3 3 Hadley So? That doesn't influence my decision. I'm not budging! #> 4 4 Mara Typical, Hadley. Stubborn as always. #> 5 5 Davis Fair enough! Let's move on. Grouping by `id` alone is actually enough, but I’ve also grouped by `name` for a convenient way to drag the name along into the summary table. [`consecutive_id()`](https://dplyr.tidyverse.org/reference/consecutive_id.html) is inspired by [`data.table::rleid()`](https://rdatatable.gitlab.io/data.table/reference/rleid.html) , which serves a similar purpose. Miscellaneous updates[](#miscellaneous-updates) ------------------------------------------------ * [`between()`](https://dplyr.tidyverse.org/reference/between.html) is no longer restricted to length 1 `left` and `right` boundaries. They are now allowed to be length 1 or the same length as `x`. Additionally, [`between()`](https://dplyr.tidyverse.org/reference/between.html) now works with any type supported by vctrs, rather than just with numerics and date-times. * [`if_else()`](https://dplyr.tidyverse.org/reference/if_else.html) has received the same updates as [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) . In particular, it is no longer as strict about typed missing values. * The ranking functions, like [`dense_rank()`](https://dplyr.tidyverse.org/reference/row_number.html) , now allow data frame inputs as a way to rank by multiple columns at once. * [`first()`](https://dplyr.tidyverse.org/reference/nth.html) , [`last()`](https://dplyr.tidyverse.org/reference/nth.html) , and [`nth()`](https://dplyr.tidyverse.org/reference/nth.html) have all gained an `na_rm` argument since they are summary functions. * [`na_if()`](https://dplyr.tidyverse.org/reference/na_if.html) now casts `y` to the type of `x` to make it clear that it is type stable on `x`. In particular, this means you can no longer do `na_if(, 0)`, which previously accidentally allowed you to attempt to replace missing values in every column with `0`. This function has always been intended as a vector function, and this is considered off-label usage. It also now replaces `NaN` values in double and complex vectors. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr-1-1-0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Dplyr-1-1-0 [![](/blog/2023/03/dplyr-1-1-1/thumbnail-sq.jpg)](/blog/2023/03/dplyr-1-1-1/) [dplyr 1.1.1](/blog/2023/03/dplyr-1-1-1/) [package](/categories/package) Davis Vaughan dplyr 1.1.1 is on CRAN! This patch release includes a number of performance regression fixes along with refinements to the multiple match join warnings that result in warnings being thrown much less often. [Read more ...](/blog/2023/03/dplyr-1-1-1/) 2023/03/22 [![](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) [dplyr 1.1.0: `pick()`, `reframe()`, and `arrange()`](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) [package](/categories/package) Davis Vaughan This final post contains a grab-bag of new features, including: `pick()` for column selection inside of data-masking functions, `reframe()` as the new home for `summarise()`'s multi-row behavior, and major performance improvements to `arrange()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/) 2023/02/07 [![](/blog/2023/02/dplyr-1-1-0-vctrs/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-vctrs/) [dplyr 1.1.0: The power of vctrs](/blog/2023/02/dplyr-1-1-0-vctrs/) [package](/categories/package) Davis Vaughan All of the dplyr vector functions, like `between()` and `case_when()`, are now powered by vctrs. We’ve also added two powerful new helpers: `case_match()` and `consecutive_id()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-vctrs/) 2023/02/02 [![](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/thumbnail-sq.jpg)](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) [dplyr 1.1.0: Per-operation grouping](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) [package](/categories/package) Davis Vaughan dplyr now supports an experimental per-operation grouping syntax. This serves as an alternative to `group_by()` and always returns an ungrouped data frame, meaning that you never need to remember to `ungroup()`. [Read more ...](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) 2023/02/01 [![](/blog/2023/01/dplyr-1-1-0-joins/thumbnail-sq.jpg)](/blog/2023/01/dplyr-1-1-0-joins/) [dplyr 1.1.0: Joins](/blog/2023/01/dplyr-1-1-0-joins/) [package](/categories/package) Davis Vaughan In dplyr 1.1.0, joins have been greatly reworked, including a new way to specify join columns, support for inequality, rolling, and overlap joins, and two new quality control arguments. [Read more ...](/blog/2023/01/dplyr-1-1-0-joins/) 2023/01/31 [![](/blog/2022/11/dplyr-1-1-0-is-coming-soon/thumbnail-sq.jpg)](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) [dplyr 1.1.0 is coming soon](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) [package](/categories/package) Davis Vaughan dplyr 1.1.0 is coming soon! This post introduces some of the exciting new features coming in 1.1.0, and includes a call-for-feedback as we finalize the release. [Read more ...](/blog/2022/11/dplyr-1-1-0-is-coming-soon/) 2022/11/28 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Writing performant code with tidy tools [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Writing performant code with tidy tools ======================================= ![](/blog/2023/04/performant-packages/thumbnail-wd.jpg) Photo by [Matt Walsh](https://unsplash.com/photos/9Q8PqfeYkMk)   2023/04/18   [package](/tags/package/) , [vctrs](/tags/vctrs/)   Simon Couch The tidyverse packages provide safe, powerful, and expressive interfaces to solve data science problems. Behind the scenes of the tidyverse is a set of lower-level tools that its developers use to build these interfaces. While these lower-level approaches are more performant than their tidy analogues, their interfaces are often less readable and safe. For most use cases in interactive data analysis, the advantages of tidyverse interfaces far outweigh the drawback in computational speed. When speed becomes an issue, though, transitioning tidy code to use these lower-level interfaces in their backend can offer substantial increases in computational performance. This post will outline alternatives to tools I love from packages like dplyr and tidyr that I use to speed up computational bottlenecks. These recommendations come from my experiences developing the [tidymodels](https://www.tidymodels.org/) packages, a collection of packages for modeling and machine learning using tidyverse principles. As such, most of these suggestions are best suited to package code, as the noted trade-off is more likely to be worth it in those settings—however, there may also be cases in analytical code, especially in production and/or with very large data sets, where these tips will be helpful. I’ve included a number of “worked examples” with each proposed alternative, showing how the tidymodels team has used these same tricks to [speed up our code](https://www.simonpcouch.com/blog/speedups-2023/) quite a bit. Before I do that, though, let’s make friends with some new R packages. Tools of the trade[](#tools-of-the-trade) ------------------------------------------ First, loading the tidyverse: library(tidyverse) The most important tools to help you understand what’s slowing your code down have little to do with the tidyverse at all! ### profvis[](#profvis) The profvis package is an R package for collecting and visualizing profiling data. library(profvis) Profiling is the process of determining how long different portions of a chunk of code take to run. For example, in this next function `slow_function()`, it’s somewhat straightforward to tell how long different portions of the following code run for if you know what [`pause()`](https://rdrr.io/pkg/profvis/man/pause.html) does. ( [`pause()`](https://rdrr.io/pkg/profvis/man/pause.html) is a function from the profvis package that just chills out for the specified amount of time. For example, `pause(1)` will wait for 1 second before finishing running.) step_1 <- function() { pause(1) } step_2 <- function() { pause(2) } slow_function <- function() { step_1() step_2() TRUE } Profiling tools would help us see that `step_1()` takes one second, while `step_2()` takes two. In practice, this is usually much harder to intuit visually. To profile code with profvis, use the [`profvis()`](https://rdrr.io/pkg/profvis/man/profvis.html) function: result <- profvis(slow_function()) Printing the `result`ing object out will visualize the time different calls within `slow_function()` took: result ![A screenshot of profvis output. A stack of grey bars sit atop a timeline that ranges from zero to three seconds. The bottom rectangle of the stack is labeled “slow_function” and stretches across the whole timeline. Two rectangles labeled “step_1” and “step_2” lie on top of the bottom rectangle, where the first stretches one-third of the way across the timeline and the second covers the remaining two-thirds.](slow-function-profvis.png) This output shows that, inside of `slow_function()`, `step_1()` took about a third of the total time and `step_2()` took two-thirds. All of the time in both of those functions was due to calling [`pause()`](https://rdrr.io/pkg/profvis/man/pause.html) . Profiling should be your first line of defense against slow-running code. Often, profiling will surface slowdowns in unexpected places, and solutions to address those slowdowns may have little to do with usage of tidy tools. To learn more about profiling, the [Measuring performance](https://adv-r.hadley.nz/perf-measure.html) chapter in Hadley Wickham’s book [Advanced R](https://adv-r.hadley.nz/) is a great place to start. ### bench[](#bench) profvis is a powerful tool to surface code slowdowns. Often, though, it may not be immediately clear how to _fix_ that slowdown. The bench package allows users to quickly test out how long different approaches to solving a problem take. For example, say we want to take the sum of the numbers in a list, but we’ve identified via profiling that this operation is slowing our code down: numbers <- as.list(1:5) numbers #> [[1]] #> [1] 1 #> #> [[2]] #> [1] 2 #> #> [[3]] #> [1] 3 #> #> [[4]] #> [1] 4 #> #> [[5]] #> [1] 5 One approach could be using the [`Reduce()`](https://rdrr.io/r/base/funprog.html) function: Reduce(sum, numbers) #> [1] 15 Another could involve converting to a vector with [`unlist()`](https://rdrr.io/r/base/unlist.html) and then using [`sum()`](https://rdrr.io/r/base/sum.html) : sum(unlist(numbers)) #> [1] 15 You may have some other ideas of how to solve this problem! How do we figure out which one is fastest, though? The [`bench::mark()`](http://bench.r-lib.org/reference/mark.html) function from bench takes in different proposals to solve the same problem and returns a tibble with information about how long they took (among other things.) res <- bench::mark( approach_1 = Reduce(sum, numbers), approach_2 = sum(unlist(numbers)) ) res %>% select(expression, median) #> # A tibble: 2 × 2 #> expression median #> #> 1 approach_1 2.25µs #> 2 approach_2 491.97ns The other nice part about [`bench::mark()`](http://bench.r-lib.org/reference/mark.html) is that it will check that each approach gives the same output, so that you don’t mistakenly compare apples and oranges. There are two important lessons to take in from this output: * The `sum(unlist())` approach was wicked fast compared to [`Reduce()`](https://rdrr.io/r/base/funprog.html) . * Both of these expressions were fast. Even the slower of the two took 2.25µs—to put that in perspective, that expression could complete 443454 iterations in a second! Keeping this bigger picture in mind is always important when benchmarking; if code runs fast enough to not be an issue in practical situations, then it need not be optimized in favor of less readable or safe code. The results of little experiments like this one can be surprising at first. Over time, though, you will develop intuition for the fastest way to solve problems you commonly solve, and will write fast code the first time around! In this case, using [`Reduce()`](https://rdrr.io/r/base/funprog.html) means calling [`sum()`](https://rdrr.io/r/base/sum.html) many times, approximately once for each element of the list, and while [`sum()`](https://rdrr.io/r/base/sum.html) isn’t particularly slow, calling an R function many times tends to have non-negligible overhead. With the `sum(unlist())` approach, there are only 2 R function calls—one for [`unlist()`](https://rdrr.io/r/base/unlist.html) and one for [`sum()`](https://rdrr.io/r/base/sum.html) —which both immediately drop into C code. ### vctrs[](#vctrs) The problems I commonly solve—and possibly you as well, as a reader of this post—often involve lots of dplyr and tidyr. When profiling the tidymodels packages, I’ve come across many places where calls to dplyr and tidyr took more time than I’d like them to, but had a lot to learn about how to speed up those operations. _Enter the vctrs package!_ library(vctrs) If you use dplyr and tidyr like I do, turns out you’re also a vctrs user! dplyr and tidyr rely on vctrs to handle all sorts of elementary operations behind the scenes, and the package is a core part of a tidy developer’s toolkit. Taken together with some functions from the tibble package, these tools provide a super efficient, albeit bare-bones, alternative interface to common data manipulation tasks like [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) ing and [`select()`](https://dplyr.tidyverse.org/reference/select.html) ing. Rewriting tidy code[](#rewriting-tidy-code) -------------------------------------------- For every performance improvement I make by rewriting dplyr and tidyr code to instead use vctrs and tibble, I make probably two or three simpler optimizations. [Tool-agnostic practices](https://adv-r.hadley.nz/perf-improve.html) such as reducing duplicated computations, implementing early returns where possible, and using vectorized implementations will likely take you far when optimizing R code. Profiling is your ground truth! When profiling indicates that otherwise well-factored code is slowed by tidy interfaces, though, all is not lost. We’ll demonstrate different ways to speed up tidy code using a version of the base R data frame `mtcars` converted to a tibble: mtcars_tbl <- as_tibble(mtcars, rownames = "make_model") mtcars_tbl #> # A tibble: 32 × 12 #> make_model mpg cyl disp hp drat wt qsec vs am gear carb #> #> 1 Mazda RX4 21 6 160 110 3.9 2.62 16.5 0 1 4 4 #> 2 Mazda RX4 … 21 6 160 110 3.9 2.88 17.0 0 1 4 4 #> 3 Datsun 710 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1 #> 4 Hornet 4 D… 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1 #> 5 Hornet Spo… 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2 #> 6 Valiant 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1 #> 7 Duster 360 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4 #> 8 Merc 240D 24.4 4 147. 62 3.69 3.19 20 1 0 4 2 #> 9 Merc 230 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2 #> 10 Merc 280 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4 #> # ℹ 22 more rows ### One-for-one replacements[](#one-for-one-replacements) Many of the core functions in dplyr have alternatives in vctrs and tibble that can be quickly transitioned. There are a couple considerations associated with each, though, and some of them make piping a bit more awkward—most of the time, when I switch these out, I remove the pipe `%>%` as well. #### `filter()`[](#filter) The dplyr code: mtcars_tbl %>% filter(hp > 100) …can be replaced by: vec_slice(mtcars_tbl, mtcars_tbl$hp > 100) Note that the second argument that determines which rows to keep requires you to actually pass the column `mtcars_tbl$hp` rather than its reference `hp`. If you feel cozier with square brackets, you can also use `[.tbl_df`:\ \ mtcars_tbl[mtcars_tbl$hp > 100, ]\ \ `[.tbl_df` is the [method for subsetting with a single square bracket when applied to tibbles](https://tibble.tidyverse.org/reference/subsetting.html)\ . Tibbles have their own methods for extracting and replacing subsets of data frames. They generally behave similarly to the analogous methods for `data.frame`s, but have small differences to improve consistency and safety.\ \ The benchmarks for these different approaches are:\ \ res <-\ bench::mark(\ dplyr = filter(mtcars_tbl, hp > 100),\ vctrs = vec_slice(mtcars_tbl, mtcars_tbl$hp > 100),\ `[.tbl_df` = mtcars_tbl[mtcars_tbl$hp > 100, ]\ ) %>%\ select(expression, median)\ \ res\ #> # A tibble: 3 × 2\ #> expression median\ #> \ #> 1 dplyr 289.93µs\ #> 2 vctrs 4.63µs\ #> 3 [.tbl_df 23.74µs\ \ \ The bigger picture of benchmarking is worth re-iterating here. While the `filter()` approach was by far the slowest expression of the three, it still only took 290µs—able to complete 3449 iterations in a second. If I’m interactively analyzing data, I won’t even notice the difference in evaluation time between these expressions, let alone care about it; the benefits of expressiveness and safety that `filter()` provide far outweigh the drawback of this slowdown. If `filter()` is called 3449 times in the backend of a machine learning pipeline, though, these alternatives may be worth transitioning to.\ \ Some examples of changes like this made to tidymodels packages: [tidymodels/parsnip#935](https://github.com/tidymodels/parsnip/pull/935)\ , [tidymodels/parsnip#933](https://github.com/tidymodels/parsnip/pull/933)\ , [tidymodels/parsnip#901](https://github.com/tidymodels/parsnip/pull/901)\ .\ \ #### `mutate()`[](#mutate)\ \ The dplyr code:\ \ mtcars_tbl <- mutate(mtcars_tbl, year = 1974L)\ \ …can be replaced by:\ \ mtcars_tbl$year <- 1974L\ \ …with benchmarks:\ \ bench::mark(\ dplyr = mutate(mtcars_tbl, year = 1974L),\ `$<-.tbl_df` = {mtcars_tbl$year <- 1974L; mtcars_tbl}\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 dplyr 302.5µs\ #> 2 $<-.tbl_df 12.8µs\ \ \ By default, both `mutate()` and `$<-.tbl_df` append the new column at the right-most position. The `.before` and `.after` arguments to `mutate()` are a really nice interface to adjust that behavior, and I miss it often when using `$<-.tbl_df`. In those cases, `select()` and its alternatives (see next section!) can be helpful.\ \ Some examples of changes like this made to tidymodels packages: [tidymodels/parsnip#933](https://github.com/tidymodels/parsnip/pull/933)\ , [tidymodels/parsnip#921](https://github.com/tidymodels/parsnip/pull/921)\ , and [tidymodels/parsnip#901](https://github.com/tidymodels/parsnip/pull/901)\ .\ \ #### `select()`[](#select)\ \ The dplyr code:\ \ select(mtcars_tbl, hp)\ \ …can be replaced by:\ \ mtcars_tbl["hp"]\ \ …with benchmarks:\ \ bench::mark(\ dplyr = select(mtcars_tbl, hp),\ `[.tbl_df` = mtcars_tbl["hp"]\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 dplyr 527.01µs\ #> 2 [.tbl_df 8.08µs\ \ \ Of course, the nice part about `select()`, and something we make use of in tidymodels quite a bit, is tidyselect. I’ve often found that we lean heavily on selecting via external vectors, i.e. character vectors, i.e. things that can be inputted to `[.tbl_df` directly. That is:\ \ cols <- c("hp", "wt")\ \ bench::mark(\ dplyr = select(mtcars_tbl, all_of(cols)),\ `[.tbl_df` = mtcars_tbl[cols]\ ) %>% \ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 dplyr 548.74µs\ #> 2 [.tbl_df 8.53µs\ \ \ Note that `[.tbl_df` always sets `drop = FALSE`.\ \ `[.tbl_df` can also be used as an alternative interface to `select()` or `relocate()` with a `.before` or `.after` argument. For instance, to place that column `year` we made in the last section as the second column, we could write:\ \ left_cols <- c("make_model", "year")\ \ mtcars_tbl[\ c(left_cols, \ setdiff(colnames(mtcars_tbl), left_cols)\ )\ ]\ \ No, thanks, but it is a good bit faster than tidyselect-based alternatives:\ \ bench::mark(\ mutate = mutate(mtcars_tbl, year = 1974L, .after = make_model),\ relocate = relocate(mtcars_tbl, year, .after = make_model),\ `[.tbl_df` = \ mtcars_tbl[\ c(left_cols, \ colnames(mtcars_tbl[!colnames(mtcars_tbl) %in% left_cols])\ )\ ],\ check = FALSE\ ) %>% \ select(expression, median)\ #> # A tibble: 3 × 2\ #> expression median\ #> \ #> 1 mutate 1.2ms\ #> 2 relocate 804.3µs\ #> 3 [.tbl_df 19.1µs\ \ \ Some examples of changes like this made to tidymodels packages: [tidymodels/parsnip#935](https://github.com/tidymodels/parsnip/pull/935)\ , [tidymodels/parsnip#933](https://github.com/tidymodels/parsnip/pull/933)\ , [tidymodels/parsnip#921](https://github.com/tidymodels/parsnip/pull/921)\ , and [tidymodels/tune#635](https://github.com/tidymodels/tune/pull/635)\ .\ \ #### `pull()`[](#pull)\ \ The dplyr code:\ \ pull(mtcars_tbl, hp)\ \ …can be replaced by:\ \ mtcars_tbl$hp\ \ …or:\ \ mtcars_tbl[["hp"]]\ \ Note that, for tibbles, `$` will raise a warning if the subsetted column doesn’t exist, while `[[` will silently return `NULL`.\ \ With benchmarks:\ \ bench::mark(\ dplyr = pull(mtcars_tbl, hp),\ `$.tbl_df` = mtcars_tbl$hp,\ `[[.tbl_df` = mtcars_tbl[["hp"]]\ ) %>%\ select(expression, median)\ #> # A tibble: 3 × 2\ #> expression median\ #> \ #> 1 dplyr 101.19µs\ #> 2 $.tbl_df 615.02ns\ #> 3 [[.tbl_df 2.25µs\ \ \ Some examples of changes like this made to tidymodels packages: [tidymodels/parsnip#935](https://github.com/tidymodels/parsnip/pull/935)\ and [tidymodels/tune#635](https://github.com/tidymodels/tune/pull/635)\ .\ \ #### `bind_*()`[](#bind_)\ \ `bind_rows()` and `bind_cols()` can be substituted for `vec_rbind()` and `vec_cbind()`, respectively. First, row-binding:\ \ bench::mark(\ dplyr = bind_rows(mtcars_tbl, mtcars_tbl),\ vctrs = vec_rbind(mtcars_tbl, mtcars_tbl)\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 dplyr 44µs\ #> 2 vctrs 14.3µs\ \ \ As for column-binding:\ \ tbl <- tibble(year = rep(1974L, nrow(mtcars_tbl)))\ \ bench::mark(\ dplyr = bind_cols(mtcars_tbl, tbl),\ vctrs = vec_cbind(mtcars_tbl, tbl)\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 dplyr 60.7µs\ #> 2 vctrs 26.2µs\ \ \ Some examples of changes like this made to tidymodels packages: [tidymodels/tune#636](https://github.com/tidymodels/tune/pull/636)\ .\ \ #### Grouping[](#grouping)\ \ In general, the introduction of groups makes these substitutions much trickier. In those cases, it’s likely best to weigh (via profiling) how significant the slowdown is and, if it’s not too bad, opt not to make any changes. For code that relies on `group_by()` and sees heavy traffic, see `vctrs::list_unchop()`, `vctrs::vec_chop()`, and `vctrs::vec_rep_each()`.\ \ ### Tibbles[](#tibbles)\ \ Tibbles are great, and I don’t want to interface with any other data frame-y thing. Some notes:\ \ * `as_tibble()` on a tibble is not “free”:\ \ bench::mark(\ on_tbl_df = as_tibble(mtcars_tbl),\ on_data.frame = as_tibble(mtcars, rownames = "make_model")\ ) %>% \ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 on_tbl_df 51.2µs\ #> 2 on_data.frame 238.6µs\ \ \ Note that the time to coerce data frames and tibbles doesn’t depend on the size of the data being coerced, in most situations.\ \ * Building a tibble from scratch using `tibble()` actually takes quite a while as well. `tibble()` handles vector recycling and name checking, builds columns sequentially, all that good stuff. If you need that, use `tibble()`, but if you’re building a tibble from well-understood inputs, use `new_tibble()`, which minimizes validation checks. For a middle ground between `tibble()` and `new_tibble(list())` in terms of both performance and safety, use the `df_list()` function from the vctrs package in place of `list()`.\ \ bench::mark(\ tibble = tibble(a = 1:2, b = 3:4),\ new_tibble_df_list = new_tibble(df_list(a = 1:2, b = 3:4), nrow = 2),\ new_tibble_list = new_tibble(list(a = 1:2, b = 3:4), nrow = 2)\ ) %>% \ select(expression, median)\ #> # A tibble: 3 × 2\ #> expression median\ #> \ #> 1 tibble 165.97µs\ #> 2 new_tibble_df_list 16.69µs\ #> 3 new_tibble_list 4.96µs\ \ \ Note that `new_tibble()` _will not check the lengths of its inputs._ Carry out simple recycling yourself, and be sure to use the `nrow` argument to get basic length checks.\ \ \ Some examples of changes like this made to tidymodels packages: [tidymodels/parsnip#945](https://github.com/tidymodels/parsnip/pull/932)\ , [tidymodels/parsnip#934](https://github.com/tidymodels/parsnip/pull/934)\ , [tidymodels/parsnip#929](https://github.com/tidymodels/parsnip/pull/929)\ , [tidymodels/parsnip#923](https://github.com/tidymodels/parsnip/pull/923)\ , [tidymodels/parsnip#902](https://github.com/tidymodels/parsnip/pull/902)\ , [tidymodels/dials#277](https://github.com/tidymodels/dials/pull/277)\ , and [tidymodels/tune#637](https://github.com/tidymodels/tune/pull/637)\ .\ \ ### Becoming join-critical[](#becoming-join-critical)\ \ Two truths:\ \ * dplyr joins are a remarkably safe and powerful way to synthesize data sources.\ \ * One ought to ask themselves “does this really need to be a join?” when combining data sources in package code.\ \ \ Some ways to intuit about join efficiency:\ \ * If this join happens multiple times, is it possible to express it as one join and then subset it when needed? i.e. if a join happens inside of a loop but the elements of the join are not indices of the loop, it’s likely possible to pull that join outside of the loop and then `vec_slice()` its results inside of the loop.\ \ * Am I using the complete outputted join result or just a portion? If I end up only making use of column names, or values in one column (as with joins approximating [lookup tables](https://adv-r.hadley.nz/subsetting.html?q=lookup#lookup-tables)\ ), or pairings between two columns, I may be able to instead use `$.tbl_df` or `[.tbl_df`.\ \ \ As an example, imagine we have another tibble that tells us additional information about the `make_model`s that I’ve driven:\ \ my_cars <- \ tibble(\ make_model = c("Honda Civic", "Subaru Forester"),\ color = c("Grey", "White")\ )\ \ my_cars\ #> # A tibble: 2 × 2\ #> make_model color\ #> \ #> 1 Honda Civic Grey \ #> 2 Subaru Forester White\ \ \ I _could_ use a join to subset down to cars in `mtcars_tbl` and add this information on the cars I’ve driven:\ \ inner_join(mtcars_tbl, my_cars, "make_model")\ #> # A tibble: 1 × 13\ #> make_model mpg cyl disp hp drat wt qsec vs am gear carb\ #> \ #> 1 Honda Civic 30.4 4 75.7 52 4.93 1.62 18.5 1 1 4 2\ #> # ℹ 1 more variable: color \ \ \ Another way to express this, though, if I can safely assume that each of my cars would have only one or zero matches in `mtcars_tbl`, is to find entries in `mtcars_tbl$make_model` that match entries in `my_cars$make_model`, subset down to those matches, and then bind columns:\ \ supplement_my_cars <- function() {\ # locate matches, assuming only 0 or 1 matches possible\ loc <- vec_match(my_cars$make_model, mtcars_tbl$make_model)\ \ # keep only the matches\ loc_mine <- which(!is.na(loc))\ loc_mtcars <- vec_slice(loc, !is.na(loc))\ \ # drop duplicated join column\ my_cars_join <- my_cars[setdiff(names(my_cars), "make_model")]\ \ vec_cbind(\ vec_slice(mtcars_tbl, loc_mtcars),\ vec_slice(my_cars_join, loc_mine)\ )\ }\ \ supplement_my_cars()\ #> # A tibble: 1 × 13\ #> make_model mpg cyl disp hp drat wt qsec vs am gear carb\ #> \ #> 1 Honda Civic 30.4 4 75.7 52 4.93 1.62 18.5 1 1 4 2\ #> # ℹ 1 more variable: color \ \ \ This is indeed quite a bit faster:\ \ bench::mark(\ inner_join = inner_join(mtcars_tbl, my_cars, "make_model"),\ manual = supplement_my_cars()\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 inner_join 438µs\ #> 2 manual 50.7µs\ \ \ At the same time, if either of these problems were even a little bit more complex, e.g. if there were possibly multiple matching `make_models` in `mtcars_tbl` or if I wanted to keep all rows in `mtcars_tbl` regardless of whether I had driven the car, then expressing this join with more bare-bones operations quickly becomes less readable and more error-prone. In those cases, too, joins in dplyr have a relatively small amount of overhead when compared to the vctrs backends underlying them. So, optimize carefully!\ \ Some examples of writing out joins in tidymodels packages: [tidymodels/parsnip#932](https://github.com/tidymodels/parsnip/pull/932)\ , [tidymodels/parsnip#931](https://github.com/tidymodels/parsnip/pull/931)\ , [tidymodels/parsnip#921](https://github.com/tidymodels/parsnip/pull/921)\ , and [tidymodels/recipes#1121](https://github.com/tidymodels/recipes/pull/1121)\ .\ \ ### `nest()`[](#nest)\ \ `nest()`s are subject to similar considerations as joins. When they allow for expressive or principled user interfaces, use them, but manipulate them sparingly in backends. Writing out `nest()` calls _can_ result in substantial speedups, though, and the process is not quite as gnarly as writing out a join. For code that relies on `nest()`s and sees heavy traffic, rewriting with vctrs may be worth the effort.\ \ For example, consider nesting `mtcars_tbl` by `cyl` and `am`:\ \ nest(mtcars_tbl, .by = c(cyl, am))\ #> # A tibble: 6 × 3\ #> cyl am data \ #> \ #> 1 6 1 \ #> 2 4 1 \ #> 3 6 0 \ #> 4 8 0 \ #> 5 4 0 \ #> 6 8 1 \ \ \ For some basic nests, `vec_split()` can do the trick.\ \ nest_cols <- c("cyl", "am")\ \ res <- \ vec_split(\ x = mtcars_tbl[setdiff(colnames(mtcars_tbl), nest_cols)],\ by = mtcars_tbl[nest_cols]\ )\ \ vec_cbind(res$key, new_tibble(list(data = res$val)))\ #> # A tibble: 6 × 3\ #> cyl am data \ #> \ #> 1 6 1 \ #> 2 4 1 \ #> 3 6 0 \ #> 4 8 0 \ #> 5 4 0 \ #> 6 8 1 \ \ \ The performance improvement in these situations can be quite substantial:\ \ bench::mark(\ nest = nest(mtcars_tbl, .by = c(cyl, am)),\ vctrs = {\ res <- \ vec_split(\ x = mtcars_tbl[setdiff(colnames(mtcars_tbl), nest_cols)],\ by = mtcars_tbl[nest_cols]\ )\ \ vec_cbind(res$key, new_tibble(list(data = res$val)))\ }\ ) %>%\ select(expression, median)\ #> # A tibble: 2 × 2\ #> expression median\ #> \ #> 1 nest 1.81ms\ #> 2 vctrs 67.61µs\ \ \ More complex nests require a good bit of facility with the vctrs package. `vec_split()`, `list_unchop()`, and `vec_chop()` are all good places to start, and these examples of writing out nests in tidymodels packages make use of other vctrs patterns: [tidymodels/tune#657](https://github.com/tidymodels/tune/pull/657)\ , [tidymodels/tune#657](https://github.com/tidymodels/tune/pull/656)\ , [tidymodels/tune#640](https://github.com/tidymodels/tune/pull/640)\ , and [tidymodels/recipes#1121](https://github.com/tidymodels/recipes/pull/1121)\ .\ \ ### Combining strings[](#combining-strings)\ \ The glue package is super helpful for writing expressive and correct strings with data, though it is quite a bit slower than `paste0()`. At the same time, `paste0()` has some tricky recycling behavior. For a middle ground in terms of both performance and safety, this short wrapper has been quite helpful:\ \ vec_paste0 <- function (...) {\ args <- vec_recycle_common(...)\ exec(paste0, !!!args)\ }\ \ name <- "Simon"\ \ bench::mark(\ glue = glue::glue("My name is {name}."),\ vec_paste0 = vec_paste0("My name is ", name, "."),\ paste0 = paste0("My name is ", name, "."),\ check = FALSE\ ) %>% \ select(expression, median)\ #> # A tibble: 3 × 2\ #> expression median\ #> \ #> 1 glue 38.99µs\ #> 2 vec_paste0 3.98µs\ #> 3 paste0 861.01ns\ \ \ My rule of thumb is to use `glue()` for errors, when the function will stop executing anyway. For simple pastes that are intended to be called repeatedly, use `vec_paste0()`. There’s a lot of gray area in between those two contexts—intuit (or profile) as you will.\ \ Wrapping up[](#wrapping-up)\ \ ----------------------------\ \ This post contains a number of tricks that offer especially performant alternatives to interfaces from dplyr and tidyr. Making use of these backend tools is certainly a trade-off; what is gained in computational performance is also offset by a decline in readability and safety, so developers ought to consider carefully when optimizations are worth the effort and risk.\ \ Thanks to Davis Vaughan for the guidance in getting started with vctrs. Also, thanks to both Davis Vaughan and Lionel Henry for their efforts in helping the tidymodels team address the bottlenecks that have been surfaced by our work on optimizations in tidyverse packages.\ \ Contents\ \ The tidyverse is proudly supported by[](https://posit.co/)\ \ [Privacy policy](https://www.tidyverse.org/google_privacy_policy)\ [](https://github.com/tidyverse)\ [](https://twitter.com/hashtag/rstats) --- # dplyr 1.1.0: Joins [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dplyr 1.1.0: Joins ================== ![](/blog/2023/01/dplyr-1-1-0-joins/thumbnail-wd.jpg) Photo by [Duy Pham](https://unsplash.com/photos/Cecb0_8Hx-o)   2023/01/31   [dplyr](/tags/dplyr/) , [dplyr-1-1-0](/tags/dplyr-1-1-0/)   Davis Vaughan [dplyr 1.1.0](https://dplyr.tidyverse.org/news/index.html#dplyr-110) is out now! This is a giant release, so we’re splitting the release announcement up into four blog posts which we’ll post over the course of this week. Today, we’re focusing on joins, including the new [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) syntax, new warnings for multiple matches, inequality joins, rolling joins, and new tools for handling unmatched rows. To learn more about joins, you might want to read the updated [joins chapter](https://r4ds.hadley.nz/joins.html) in the upcoming 2nd edition of [R for Data Science](https://r4ds.hadley.nz) . This version of dplyr includes a number of features inspired by our [data.table](https://cran.r-project.org/web/packages/data.table/index.html) friends. The inequality and rolling joins we discuss today were popularized in R by data.table, and greatly inspired our own implementation. To see the other blog posts in this series, head [here](https://www.tidyverse.org/tags/dplyr-1-1-0/) . You can install it from CRAN with: install.packages("dplyr") library(dplyr) `join_by()`[](#join_by) ------------------------ Consider the following two tables, `transactions` and `companies`. `transactions` tracks sales across various years for different companies, and `companies` connects the short company id to its actual company name - either Patagonia (a fellow B-Corp!) or RStudio. transactions <- tibble( company = c("A", "A", "B", "B"), year = c(2019, 2020, 2021, 2023), revenue = c(50, 4, 10, 12) ) transactions #> # A tibble: 4 × 3 #> company year revenue #> #> 1 A 2019 50 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 12 companies <- tibble( id = c("A", "B"), name = c("Patagonia", "RStudio") ) companies #> # A tibble: 2 × 2 #> id name #> #> 1 A Patagonia #> 2 B RStudio To join these two tables together, we might use an inner join: transactions |> inner_join(companies, by = c(company = "id")) #> # A tibble: 4 × 4 #> company year revenue name #> #> 1 A 2019 50 Patagonia #> 2 A 2020 4 Patagonia #> 3 B 2021 10 RStudio #> 4 B 2023 12 RStudio This works great, but has always felt a little clunky. Specifying `c(company = "id")` is a little awkward because it uses `=`, not `==`: here we’re asserting that we want `company` to equal `id`, not naming a function argument or performing assignment. We’ve improved on this with a new helper, [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) , which takes expressions in a way that allows you to more naturally express this join: join_by(company == id) #> Join By: #> - company == id This _join specification_ can be used as the `by` argument in any of the `*_join()` functions: transactions |> inner_join(companies, by = join_by(company == id)) #> # A tibble: 4 × 4 #> company year revenue name #> #> 1 A 2019 50 Patagonia #> 2 A 2020 4 Patagonia #> 3 B 2021 10 RStudio #> 4 B 2023 12 RStudio This small quality of life improvement is just one of the many new features that come with [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) . We’ll look at more of these next. Multiple matches[](#multiple-matches) -------------------------------------- * * * **Update**: As of March 22, dplyr 1.1.1 is available on CRAN, which alters the behavior of multiple match detection so that you see warnings much less often. Read [all about it](/blog/2023/03/dplyr-1-1-1/) or install it now with `install.packages("dplyr")`. * * * To make things a little more interesting, we’ll add one more column to `companies`, and one more row: companies <- tibble( id = c("A", "B", "B"), since = c(1973, 2009, 2022), name = c("Patagonia", "RStudio", "Posit") ) companies #> # A tibble: 3 × 3 #> id since name #> #> 1 A 1973 Patagonia #> 2 B 2009 RStudio #> 3 B 2022 Posit This table now also tracks name changes that have happened over the course of a company’s history. In 2022, we changed our name from RStudio to Posit, so we’ve tracked that as an additional row in our dataset. Note that both RStudio and Posit are given an `id` of `B`, which links back to the `transactions` table. If we were to join these two tables together, ideally we’d bring over the name that was in effect when the transaction took place. For example, for the transaction in 2021, the company was still RStudio, so ideally we’d only match up against the RStudio row in `companies`. If we colored the expected matches, they’d look something like this: ![](img/ideal-join.png) How can we do this? We can try the same join from before, but we won’t like the results: faulty <- transactions |> inner_join(companies, by = join_by(company == id)) #> Warning in inner_join(transactions, companies, by = join_by(company == id)): Each row in `x` is expected to match at most 1 row in `y`. #> ℹ Row 3 of `x` matches multiple rows. #> ℹ If multiple matches are expected, set `multiple = "all"` to silence this #> warning. faulty #> # A tibble: 6 × 5 #> company year revenue since name #> #> 1 A 2019 50 1973 Patagonia #> 2 A 2020 4 1973 Patagonia #> 3 B 2021 10 2009 RStudio #> 4 B 2021 10 2022 Posit #> 5 B 2023 12 2009 RStudio #> 6 B 2023 12 2022 Posit Company `A` matches correctly, but since we only joined on the company id, we get _multiple matches_ for each of company `B`'s transactions and end up with more rows than we started with. This is a problem, as we were expecting a 1:1 match for each row in `transactions`. Multiple matches in equality joins like this one are typically unexpected (even though they are baked in to SQL) so we’ve also added a new warning to alert you when this happens. If multiple matches are expected, you can explicitly set `multiple = "all"` to silence this warning. This also serves as a code “sign post” for future readers of your code to let them know that this is a join that is expected to increase the number of rows in the data. If multiple matches _aren’t_ expected, you can also set `multiple = "error"` to immediately halt the analysis. We expect this will be useful as a quality control check for production code where you might rerun analyses with new data on a rolling basis. Inequality joins[](#inequality-joins) -------------------------------------- To actually fix this issue, we’ll need to expand our join specification to include another condition. Let’s zoom in to just 2021: filter(faulty, company == "B", year == 2021) #> # A tibble: 2 × 5 #> company year revenue since name #> #> 1 B 2021 10 2009 RStudio #> 2 B 2021 10 2022 Posit We want to retain the match with RStudio, but not with Posit (because the name hasn’t changed yet). One way to express this is by using the `year` and `since` columns to state that you only want a match if the transaction `year` occurred _after_ a name change: # `year[i] >= since`? 2021 >= 2009 #> [1] TRUE 2021 >= 2022 #> [1] FALSE Because [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) accepts expressions, we can express this inequality directly inside the join specification: join_by(company == id, year >= since) #> Join By: #> - company == id #> - year >= since transactions |> inner_join(companies, join_by(company == id, year >= since)) #> # A tibble: 5 × 5 #> company year revenue since name #> #> 1 A 2019 50 1973 Patagonia #> 2 A 2020 4 1973 Patagonia #> 3 B 2021 10 2009 RStudio #> 4 B 2023 12 2009 RStudio #> 5 B 2023 12 2022 Posit This eliminated the 2021 match to Posit, as expected! This type of join is known as an _inequality join_, i.e. it involves at least one join expression containing one of the following inequality conditions: `>=`, `>`, `<=`, or `<`. However, we still have 2 matches corresponding to the 2023 year. In this case, we only wanted the match to Posit. We can understand why we are still getting multiple matches here by running the same row-by-row analysis as before: # `year[i] >= since`? Both are true! 2023 >= 2009 #> [1] TRUE 2023 >= 2022 #> [1] TRUE To remove the last problematic match of the 2023 transaction to the RStudio name, we’ll need to refine our join specification one more time. Rolling joins[](#rolling-joins) -------------------------------- Inequality conditions like `year >= since` are powerful, but since the condition is only bounded on one side it is common for them to return a large number of matches. Since multiple matches are the typical case with inequality joins, we don’t get a warning like with the equality join, but we clearly still haven’t gotten the join right. As a reminder, here are where we still have too many matches: transactions |> inner_join(companies, join_by(company == id, year >= since)) |> filter(company == "B", year == 2023) #> # A tibble: 2 × 5 #> company year revenue since name #> #> 1 B 2023 12 2009 RStudio #> 2 B 2023 12 2022 Posit We need a way to filter down the matches returned from `year >= since` to only the most recent name change. In other words, we prefer the Posit match over the RStudio match because 2022 is _closer_ to the transaction year of 2023 than 2009 is. We can express this in [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) by using a helper named `closest()`. transactions |> inner_join(companies, join_by(company == id, closest(year >= since))) #> # A tibble: 4 × 5 #> company year revenue since name #> #> 1 A 2019 50 1973 Patagonia #> 2 A 2020 4 1973 Patagonia #> 3 B 2021 10 2009 RStudio #> 4 B 2023 12 2022 Posit `closest(year >= since)` finds all of the matches in `since` for a particular `year`, and then filters them down to only the closest match to that `year`. This is known as a _rolling join_, because in this case it _rolls_ the most recent name change forward to match up with the transaction. Rolling joins were popularized by data.table, and are related to `ASOF` joins supported by some SQL flavors. There is a third new class of joins supported by [`join_by()`](https://dplyr.tidyverse.org/reference/join_by.html) that we won’t discuss today known as _overlap joins_. These are particularly useful in time series where you are looking for cases where a date or range of dates from one table _overlaps_ a range of dates in another table. There are three helpers for overlap joins: [`between()`](https://dplyr.tidyverse.org/reference/join_by.html#overlap-joins) , [`overlaps()`](https://dplyr.tidyverse.org/reference/join_by.html#overlap-joins) , and [`within()`](https://dplyr.tidyverse.org/reference/join_by.html#overlap-joins) , which you can read more about [in the documentation](https://dplyr.tidyverse.org/reference/join_by.html#overlap-joins) . Unmatched rows[](#unmatched-rows) ---------------------------------- I mentioned earlier that we expected a 1:1 match between `transactions` and `companies`. We saw that `multiple` can help protect us from having too many matches, but what about not having enough? Consider what happens if we add a new company to `transactions` without a corresponding match in `companies`. transactions <- transactions |> tibble::add_row(company = "C", year = 2023, revenue = 15) transactions #> # A tibble: 5 × 3 #> company year revenue #> #> 1 A 2019 50 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 12 #> 5 C 2023 15 transactions |> inner_join( companies, join_by(company == id, closest(year >= since)) ) #> # A tibble: 4 × 5 #> company year revenue since name #> #> 1 A 2019 50 1973 Patagonia #> 2 A 2020 4 1973 Patagonia #> 3 B 2021 10 2009 RStudio #> 4 B 2023 12 2022 Posit We’ve accidentally lost the `C` row! If you don’t expect any unmatched rows, you can now catch this problem automatically by using our other new quality control argument, `unmatched`: transactions |> inner_join( companies, join_by(company == id, closest(year >= since)), unmatched = "error" ) #> Error in `inner_join()`: #> ! Each row of `x` must have a match in `y`. #> ℹ Row 5 of `x` does not have a match. If you’ve been questioning why I’ve been using an [`inner_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) over a [`left_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) this whole time, `unmatched` is why. We could use a [`left_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) : transactions |> left_join( companies, join_by(company == id, closest(year >= since)), unmatched = "error" ) #> # A tibble: 5 × 5 #> company year revenue since name #> #> 1 A 2019 50 1973 Patagonia #> 2 A 2020 4 1973 Patagonia #> 3 B 2021 10 2009 RStudio #> 4 B 2023 12 2022 Posit #> 5 C 2023 15 NA NA But you’ll notice that we don’t get an error here. `unmatched` will only error if the input that has the potential to drop rows has an unmatched row. The reason you’d use a [`left_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) is to ensure that rows from `x` are always retained, so it wouldn’t make sense to error when rows from `x` are also unmatched. If `y` had unmatched rows instead, _then_ it would have errored because those rows would otherwise be lost from the join. In an [`inner_join()`](https://dplyr.tidyverse.org/reference/mutate-joins.html) , both inputs can potentially drop rows, so `unmatched = "error"` checks for unmatched rows in both inputs. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dtplyr 1.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dtplyr 1.3.0 ============ ![](/blog/2023/02/dtplyr-1-3-0/thumbnail-wd.jpg) Photo by [Neil Cooper](https://unsplash.com/photos/uwI8R_FyLrI)   2023/02/24   [dplyr](/tags/dplyr/) , [dtplyr](/tags/dtplyr/)   Hadley Wickham We’re thrilled to announce the release of [dtplyr](https://dtplyr.tidyverse.org) 1.3.0. dtplyr gives you the speed of [data.table](http://r-datatable.com/) with the syntax of dplyr; you write dplyr (and tidyr) code and dtplyr translates it to the data.table equivalent. You can install it from CRAN with: install.packages("dtplyr") This blog post will give you an overview of the changes in this version: dtplyr no longer adds translations directly to data.tables, it includes some dplyr 1.1.0 updates, and we have made some performance improvements. As always, you can see a full list of changes in the [release notes](https://github.com/tidyverse/dtplyr/releases/tag/v1.3.0) library(dtplyr) library(dplyr, warn.conflicts = FALSE) Breaking changes[](#breaking-changes) -------------------------------------- In previous versions, dtplyr registered translations that kicked in whenever you used a data.table. This [caused problems](https://github.com/tidyverse/dtplyr/issues/312) because merely loading dtplyr could cause otherwise ok code to fail because dplyr and tidyr functions would now return `lazy_dt` objects instead of `data.table` objects. To avoid this problem, we have removed those S3 methods so you must now explicitly opt-in to dtplyr translations by using [`lazy_dt()`](https://dtplyr.tidyverse.org/reference/lazy_dt.html) . dplyr 1.1.0[](#dplyr-110) -------------------------- This release brings support for dplyr 1.1.0’s [per-operation grouping](https://www.tidyverse.org/blog/2023/02/dplyr-1-1-0-per-operation-grouping/) and [`pick()`](https://dplyr.tidyverse.org/reference/pick.html) : dt <- lazy_dt(data.frame(x = 1:10, id = 1:2)) dt |> summarise(mean = mean(x), .by = id) |> show_query() #> `_DT1`[, .(mean = mean(x)), keyby = .(id)] dt <- lazy_dt(data.frame(x = 1:10, y = runif(10))) dt |> mutate(row_sum = rowSums(pick(x))) |> show_query() #> copy(`_DT2`)[, `:=`(row_sum = rowSums(data.table(x = x)))] Per-operation grouping was one of the dplyr 1.1.0 features inspired by data.table, so it’s neat to see it come full circle in this dtplyr release. Future releases will add support for other dplyr 1.1.0 features like the new [`join_by()`](https://www.tidyverse.org/blog/2023/01/dplyr-1-1-0-joins/#join_by) syntax and [`reframe()`](https://www.tidyverse.org/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/#reframe) . Improved translations[](#improved-translations) ------------------------------------------------ dtplyr gains new translations for [`add_count()`](https://dplyr.tidyverse.org/reference/count.html) and `unite()`, and the ranking functions, [`min_rank()`](https://dplyr.tidyverse.org/reference/row_number.html) , [`dense_rank()`](https://dplyr.tidyverse.org/reference/row_number.html) , [`percent_rank()`](https://dplyr.tidyverse.org/reference/percent_rank.html) , & [`cume_dist()`](https://dplyr.tidyverse.org/reference/percent_rank.html) are now mapped to their `data.table` equivalents: dt |> add_count() |> show_query() #> copy(`_DT2`)[, `:=`(n = .N)] dt |> tidyr::unite("z", c(x, y)) |> show_query() #> copy(`_DT2`)[, `:=`(z = paste(x, y, sep = "_"))][, `:=`(c("x", \ #> "y"), NULL)] dt |> mutate(r = min_rank(x)) |> show_query() #> copy(`_DT2`)[, `:=`(r = frank(x, ties.method = "min", na.last = "keep"))] dt |> mutate(r = dense_rank(x)) |> show_query() #> copy(`_DT2`)[, `:=`(r = frank(x, ties.method = "dense", na.last = "keep"))] This release also includes three translation improvements that yield better performance. When data has previously been copied [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) will use `setorder()` instead of [`order()`](https://rdrr.io/r/base/order.html) and [`select()`](https://dplyr.tidyverse.org/reference/select.html) will drop unwanted columns by reference (i.e. with `var := NULL`). And [`slice()`](https://dplyr.tidyverse.org/reference/slice.html) now uses an intermediate variable to reduce computation time of row selection. Acknowledgements[](#acknowledgements) -------------------------------------- A massive thanks to [Mark Fairbanks](https://github.com/markfairbanks) who did most of the work for this release, ably aided by the other dtplyr maintainers [@eutwt](https://github.com/eutwt) and [Maximilian Girlich](https://github.com/mgirlich) . And thanks to everyone else who helped make this release possible, whether it was with code, documentation, or insightful comments: [@abalter](https://github.com/abalter) , [@akaviaLab](https://github.com/akaviaLab) , [@camnesia](https://github.com/camnesia) , [@caparks2](https://github.com/caparks2) , [@DavisVaughan](https://github.com/DavisVaughan) , [@eipi10](https://github.com/eipi10) , [@hadley](https://github.com/hadley) , [@jmbarbone](https://github.com/jmbarbone) , [@johnF-moore](https://github.com/johnF-moore) , [@lschneiderbauer](https://github.com/lschneiderbauer) , and [@NicChr](https://github.com/NicChr) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # forcats [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Forcats [![](/blog/2023/01/forcats-1-0-0/thumbnail-sq.jpg)](/blog/2023/01/forcats-1-0-0/) [forcats 1.0.0](/blog/2023/01/forcats-1-0-0/) [package](/categories/package) Hadley Wickham There are no major new features in this version of forcats, but the 1.0.0 label now clearly advertises that this a stable member of the tidyverse. [Read more ...](/blog/2023/01/forcats-1-0-0/) 2023/01/30 [![](/blog/2020/03/forcats-0-5-0/thumbnail-sq.jpg)](/blog/2020/03/forcats-0-5-0/) [forcats 0.5.0](/blog/2020/03/forcats-0-5-0/) [package](/categories/package) Mara Averick Announcing the release of forcats 0.5.0 on CRAN. [Read more ...](/blog/2020/03/forcats-0-5-0/) 2020/03/02 [![](/blog/2019/05/forcats-0-4-0/forcats-0-4-0-sq.jpg)](/blog/2019/05/forcats-0-4-0/) [forcats 0.4.0](/blog/2019/05/forcats-0-4-0/) [package](/categories/package) Mara Averick What’s new in forcats 0.4.0? [Read more ...](/blog/2019/05/forcats-0-4-0/) 2019/05/10 [![](/blog/2018/02/forcats-0-3-0/forcats-0-3-0-sq.jpg)](/blog/2018/02/forcats-0-3-0/) [forcats 0.3.0](/blog/2018/02/forcats-0-3-0/) [package](/categories/package) Mara Averick A new version of forcats is now on CRAN. [Read more ...](/blog/2018/02/forcats-0-3-0/) 2018/02/21 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # webR 0.1.0 has been released [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) webR 0.1.0 has been released ============================ ![](/blog/2023/03/webr-0-1-0/thumbnail-wd.jpg) Photo by [DALL·E 2](https://openai.com/dall-e-2/)   2023/03/09   [webr](/tags/webr/) , [wasm](/tags/wasm/) , [webassembly](/tags/webassembly/)   George Stagg We’re super excited to announce the release of webR v0.1.0! This is the first release of webR intended for general use by the web development and R communities and is the result of almost a year of hard work by the webR developers. This post will introduce webR, demonstrate some of the possibilities that running R in a web browser brings, and give a quick overview of how to include webR in your own TypeScript or JavaScript web applications. Introduction[](#introduction) ------------------------------ WebR is a version of the open-source R interpreter compiled for WebAssembly, along with a supporting TypeScript library for interacting with the console and R objects from a JavaScript environment. By compiling R to WebAssembly a user can visit a website and run R code directly within the web browser, without R installed on their device or a supporting computational R server. All that is required is a normal web server, including the type of cloud hosting service provided by Github Pages or Netlify. How it works[](#how-it-works) ------------------------------ WebR’s core is based around compiling the open-source R interpreter for [WebAssembly](https://webassembly.org) , using the [Emscripten](https://emscripten.org) compiler suite along with [LLVM Flang](https://flang.llvm.org/docs/) to work with R’s pre-existing C and Fortran based source code. WebAssembly (often abbreviated as Wasm) is a standard defining a virtual stack machine along with a corresponding _bytecode_. Efficient Wasm engines have already been implemented in most modern web browsers, which allows for the deployment of high performance Wasm applications on the web. While it’s certainly possible for an interested programmer to write Wasm bytecode by hand, it is not a requirement to do so. Similar to how code and data is compiled into _machine code_ for a certain computer processor, code and data can be compiled into the Wasm bytecode by compiler software that supports the Wasm standard. However, unlike with traditional machine code, the Wasm virtual machine (VM) is consistent across multiple different types of environment, architecture, and device – in theory the same bytecode binary can run anywhere without having to be recompiled for that environment. In this way the Wasm VM is similar to Java’s JVM. However, in comparison to the JVM, Wasm has been designed and built from the ground up for use on the modern web, requiring strict sandboxing and security controls. Future use for WebAssembly has also been identified in server-side web development, containerisation, cloud computing, and more. With these applications, Wasm has been suggested as a universal binary format of the future. Multiple implementations of the Wasm VM already exist designed to run _outside_ a web browser, through proposed Wasm standards such as [WASI](https://wasi.dev) . What’s possible?[](#whats-possible) ------------------------------------ Undoubtedly, webR opens a world of possibilities for the interactive use of R and data science on the web. ### An online R console[](#an-online-r-console) A web-based interactive R console is included in the webR source repository as a demonstration of integrating webR into a wider web application. A publicly accessible instance of the webR console can be found at [https://webr.r-wasm.org/latest/](https://webr.r-wasm.org/latest/) . [![A screenshot showing the demo webR console creating a plot](webr-repl.png)](webr-repl.png) With the webR online console a new user can get up and running with R in seconds. The webR console is also functional on many modern mobile devices, where traditional versions of R are not always available for installation at all[1](#fn:1) . It’s possible to perform data analysis on reasonably large datasets by uploading data to a Virtual File System (VFS). The webR console provides an interface to view and interact with the VFS (**Files** tab, top right). Once a data file has been uploaded to the VFS it can be read by R like any standard file. [![A screenshot showing the demo webR console loading a data file](webr-repl2.png)](webr-repl2.png) Note that uploading and downloading files to the VFS in this way does not actually involve transferring any data over the network. However, webR has been built so that it is possible to load data into webR over the network by using R’s built in functions that can download from URL, such as [`read.csv()`](https://rdrr.io/r/utils/read.table.html) [2](#fn:2) . [![A screenshot showing the demo webR console loading a data file from URL](webr-repl3.png)](webr-repl3.png) Plotting is also supported (**Plotting** tab, top right), meaning a user can produce beautiful plot output with the webR console, closing the loop of reading data, performing analysis, and producing output. It is entirely feasible that a casual user could perform the basics of data science entirely within their web browser using webR. ### An educational tool[](#an-educational-tool) Consider the following code block containing some simple R code. After a short loading period while the webR binary and supporting files are downloaded, a **Run code** button is enabled on the code block, with the code itself able to be edited and remixed on the fly. Feel free to try this out now! Loading webR... After executing the R code once, try changing the `am` variable in the model to `gear` and then clicking **Run code** again. You should immediately see how changing the model affects the components of the resulting fit. There is a real R session running and powering this code block – try replacing the entire code with something new! The following interactive code block produces an R plot that is directly embedded into the page. As with the previous example, the plot can be recreated or remixed multiple times by the reader simply by clicking the **Run Code** button. Loading webR... In my experience this way of interacting and experimenting with R code without the mental overhead of context switching from a web browser to an R console, or copying and pasting lines of example code, feels extremely fresh and exciting. An exciting potential application for webR is providing high-quality educational web content in exactly this kind of format. ### Reproducible reports[](#reproducible-reports) A core principle of good science is that results should be repeatable and reproducible by others. Unfortunately the misuse of data analysis, leading to unreliable results, [is a known issue](https://en.wikipedia.org/wiki/Misuse_of_statistics) . The idea of a reproducible report is to bring the philosophy of repeatability to the delivery format itself. Reproducible reports weave together explanatory prose, data science, source code, output and figures; all in a single place with a consistent execution environment. With this, a user reading the report has everything they need to reproduce and confirm results for themselves. While Jupyter notebooks were not the first implementation of executable documents[3](#fn:3) , their popularity has grown over the last decade or so as a way to support high quality reproducible reports. Jupyter has been named [_“The data scientists’ computational notebook of choice”_](https://www.nature.com/articles/d41586-018-07196-1) and almost 10 million Jupyter notebooks were publicly accessible on GitHub as of Oct 2020[4](#fn:4) . While a Jupyter notebook usually requires a Python and Jupyter installation to fully reproduce results, recent work by the [JupyterLite](https://jupyterlite.readthedocs.io/en/latest/) team uses Wasm to bring Jupyter to the web browser. JupyterLite can be used with [Pyodide](https://pyodide.org/en/stable/) to run Python based notebooks directly in the browser. WebR aims to provide that same experience for Jupyter notebooks based on R. As part of the initial release of webR, we are also releasing a [webR kernel for JupyterLite](https://github.com/r-wasm/jupyterlite-webr-kernel) , allowing users to write and execute reproducible Jupyter notebooks for R directly in the web browser. A JupyterLite instance with the webR kernel available can be found at [https://jupyter.r-wasm.org/](https://jupyter.r-wasm.org/) , along with a sample R Jupyter notebook demonstrating a reproducible report. [![A screenshot showing the webR JupyterLite kernel](jupyter.png)](jupyter.png) The JupyterLite kernel for R is still in the early stages of development and [includes some limitations](https://github.com/r-wasm/jupyterlite-webr-kernel#limitations) , but the core infrastructure is in place with the release of webR. ### R packages[](#r-packages) R has a rich history of user-created extensions through the use of R packages. Most packages are a combination of R and C or C++ code, and so many packages must be compiled from source for the system they are running on. Unfortunately, it is not possible to install packages in this way in webR. Such an installation process would require an entire C/C++ to WebAssembly compiler toolchain running in the web page! For the moment, downloading pre-compiled Wasm binaries is the only supported way to install packages in webR. A pre-installed `webr` support package provides a helper function `webr::install()` which can be used to install packages from a CRAN-like repository. As part of the webR release we have provided a small repository of binary R packages compiled for Wasm, publicly hosted with URL `https://repo.r-wasm.org/`. Using webR in your own projects[](#using-webr-in-your-own-projects) -------------------------------------------------------------------- WebR aims to be as quick and easy to use as possible for those familiar with JavaScript web development. While a short introduction to using webR follows in this blog post, we think the best way to get up and running is by reading the Getting Started section of the [webR documentation](https://docs.r-wasm.org/webr/latest/) . The documentation goes into further detail about how to download webR, technical requirements for serving web pages that use webR, and provides more detailed examples. ### Downloading and using webR from npm[](#downloading-and-using-webr-from-npm) For a project with dependencies managed by npm, the [webR JavaScript package](https://www.npmjs.com/package/@r-wasm/webr) can be installed by using the command, npm i @r-wasm/webr Once available, webR can be imported into a project and a new instance of webR initialised with, import { WebR } from '@r-wasm/webr'; const webR = new WebR(); await webR.init(); Once a new instance of the `WebR()` class has been created, webR will begin to download WebAssembly binaries from the public CDN, and R will be started. ### Downloading webR release packages[](#downloading-webr-release-packages) Full release packages for webR can also be downloaded from the webR [GitHub Releases](https://github.com/r-wasm/webR/releases) page. The full release packages include the webR JavaScript loader, along with WebAssembly binaries for R and its supporting libraries. Hosting a full release package on a web server makes it possible to use webR entirely on your own infrastructure, rather than relying on downloading Wasm binaries from the public CDN. ### An example of executing R code[](#an-example-of-executing-r-code) Once R is ready, the JavaScript promise returned by `webR.init()` will resolve. At this point R code can be evaluated and results converted into JavaScript objects, let result = await webR.evalR('rnorm(10,5,1)'); let output = await result.toArray(); console.log(output); In the above example the `result` object can be thought of as a reference to a specific R object, and is converted into a standard JavaScript array using the `toArray()` function. Further examples and details of how to interact with the R console and work with R objects can be found in the [webR documentation](https://docs.r-wasm.org/webr/latest/examples.html) . The future of webR[](#the-future-of-webr) ------------------------------------------ Going forward we plan to expand and improve webR, including compiling more R packages for the webR public package repository. It is our hope that we can provide the same web-based computational infrastructure for R that [Pyodide](https://pyodide.org/en/stable/) has provided for the Python ecosystem. While WebAssembly engines are in theory able to provide near-native performance, when it comes to the requirements for advanced data science or the deployment of sophisticated machine learning models, the benefits of running tools such as the RStudio IDE natively or a high-performance cloud deployment will likely always outperform the relatively restricted WebAssembly virtual machine. Despite this, webR can provide a smooth, interactive and immediate introduction to the world of working with data in R. Users who have not had the chance to use R in the past due to the barriers raised by the installation of new software to their workstation, or registration for a cloud-based service, might yet still be convinced to introduce R to their workflow though an introduction with interactive examples or short reports powered by webR. The opportunity for enhancing educational content also continues beyond introductory materials. Many R packages are documented online, using automated tools such as [pkgdown](https://pkgdown.r-lib.org) to produce a dedicated website for the package. Alongside an introductory description, package websites usually also include usage details in the form of example code, reference documentation, and vignette articles. However, if a potential user would like to try the package for themselves, often the only way is by installing the package onto their own machine. Immediately interactive examples, powered by webR, are an interesting future possibility that would reduce this kind of barrier to entry. Fairly recently, the Shiny team announced [Shiny for Python](https://shiny.rstudio.com/py/) , a feature rich reactive web application framework targeting Python. Of particular note, the team used WebAssembly and Pyodide as a way to run a [Shinylive](https://shiny.rstudio.com/py/docs/shinylive.html) server directly in the user’s web browser. One of the most exciting possible applications for webR is a similar architecture targeting the traditional R version of Shiny. Is it possible for a _Shinylive for R_ to be powered by webR? We certainly hope so. Acknowledgements[](#acknowledgements) -------------------------------------- A massive thank you to all early webR users for their willingness to experiment and their feedback in the form of GitHub issues and pull requests, [@Anurodhyadav](https://github.com/Anurodhyadav) , [@barryrowlingson](https://github.com/barryrowlingson) , [@christianp](https://github.com/christianp) , [@ekianjo](https://github.com/ekianjo) , [@georgestagg](https://github.com/georgestagg) , [@HTUser-1](https://github.com/HTUser-1) , [@jason-variadiclabs](https://github.com/jason-variadiclabs) , [@jjesusfilho](https://github.com/jjesusfilho) , [@kdpsingh](https://github.com/kdpsingh) , [@lionel-](https://github.com/lionel-) , [@psychemedia](https://github.com/psychemedia) , [@Sjesc](https://github.com/Sjesc) , [@SugarRayLua](https://github.com/SugarRayLua) [@unclecode](https://github.com/unclecode) , and [@wch](https://github.com/wch) . * * * 1. I am aware of at least one early adopter using webR as a way to access R on their Apple iPad. [↩︎](#fnref:1) 2. Note that there are some security measures in place when fetching data that are applied to all web applications. Downloading datasets from URL requires that the web server providing the data supports and allows [Cross Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) [↩︎](#fnref:2) 3. Knuth originally introduced the precursor [Literate Programming](https://en.wikipedia.org/wiki/Literate_programming) paradigm in 1984, and more recently tools such as Sweave, knitr and RMarkdown enable embedding R and computational results directly into a report. [↩︎](#fnref:3) 4. Admittedly, only a small proportion using an R kernel. [The overwhelming majority use Python, R comes second, and Julia third.](https://blog.jetbrains.com/datalore/2020/12/17/we-downloaded-10-000-000-jupyter-notebooks-from-github-this-is-what-we-learned/) [↩︎](#fnref:4) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr 1.1.0: `pick()`, `reframe()`, and `arrange()` [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dplyr 1.1.0: `pick()`, `reframe()`, and `arrange()` =================================================== ![](/blog/2023/02/dplyr-1-1-0-pick-reframe-arrange/thumbnail-wd.jpg) Photo by [Priscilla Du Preez](https://unsplash.com/photos/XgoHMMkE02I)   2023/02/07   [dplyr](/tags/dplyr/) , [dplyr-1-1-0](/tags/dplyr-1-1-0/)   Davis Vaughan In this final [dplyr 1.1.0](https://dplyr.tidyverse.org/news/index.html#dplyr-110) post, we’ll take a look at two new verbs, [`pick()`](https://dplyr.tidyverse.org/reference/pick.html) and [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) , along with some changes to [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) that improve both reproducibility and performance. If you missed our previous posts, you should definitely go back and [check them out](https://www.tidyverse.org/tags/dplyr-1-1-0/) ! You can install it from CRAN with: install.packages("dplyr") library(dplyr) set.seed(12345) `pick()`[](#pick) ------------------ One thing we noticed after dplyr 1.0.0 was released is that many people like to use [`across()`](https://dplyr.tidyverse.org/reference/across.html) for its column selection features while working inside a data-masking function like [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) or [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) . This is typically useful if you have a function that takes data frames as inputs, or if you need to compute features about a specific subset of columns. df <- tibble( x_1 = c(1, 3, 2, 1, 2), x_2 = 6:10, w_4 = 11:15, y_2 = c(5, 2, 4, 0, 6) ) df |> summarise( n_x = ncol(across(starts_with("x"))), n_y = ncol(across(starts_with("y"))) ) #> # A tibble: 1 × 2 #> n_x n_y #> #> 1 2 1 [`across()`](https://dplyr.tidyverse.org/reference/across.html) is intended to apply a function to each of these columns, rather than just select them, which is why its name doesn’t feel natural for this operation. In dplyr 1.1.0 we’ve introduced [`pick()`](https://dplyr.tidyverse.org/reference/pick.html) , a specialized column selection variant with a more natural name: df |> summarise( n_x = ncol(pick(starts_with("x"))), n_y = ncol(pick(starts_with("y"))) ) #> # A tibble: 1 × 2 #> n_x n_y #> #> 1 2 1 [`pick()`](https://dplyr.tidyverse.org/reference/pick.html) is particularly useful in combination with ranking functions like [`dense_rank()`](https://dplyr.tidyverse.org/reference/row_number.html) , which have been upgraded in 1.1.0 to take data frames as inputs, serving as a way to jointly rank by multiple columns at once. df |> mutate( rank1 = dense_rank(x_1), rank2 = dense_rank(pick(x_1, y_2)) # Using `y_2` to break ties in `x_1` ) #> # A tibble: 5 × 6 #> x_1 x_2 w_4 y_2 rank1 rank2 #> #> 1 1 6 11 5 1 2 #> 2 3 7 12 2 3 5 #> 3 2 8 13 4 2 3 #> 4 1 9 14 0 1 1 #> 5 2 10 15 6 2 4 We haven’t deprecated using [`across()`](https://dplyr.tidyverse.org/reference/across.html) without supplying `.fns` yet, but we plan to in the future now that [`pick()`](https://dplyr.tidyverse.org/reference/pick.html) exists as a better alternative. `reframe()`[](#reframe) ------------------------ As we mentioned in the [coming soon](https://www.tidyverse.org/blog/2022/11/dplyr-1-1-0-is-coming-soon/) blog post, in dplyr 1.1.0 we’ve decided to walk back the change we introduced to [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) in dplyr 1.0.0 that allowed it to return per-group results of any length, rather than results of length 1. We think that the idea of multi-row results is extremely powerful, as it serves as a flexible way to apply arbitrary operations to each group, but we’ve realized that [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) wasn’t the best home for it because it increases the chance for users to run into silent recycling bugs (thanks to [Kirill Müller](https://github.com/tidyverse/dplyr/issues/6382) and [David Robinson](https://twitter.com/drob/status/1563198515626770432?s=20&t=iTFWSCPNOGWalIrpXHx2qg) for bringing this to our attention). As an example, here we’re computing the mean and standard deviation of `x`, grouped by `g`. Unfortunately, I accidentally forgot to use `sd(x)` and instead just typed `x`. Because of how [tidyverse recycling rules](https://vctrs.r-lib.org/reference/vector_recycling_rules.html) work, the multi-row behavior silently recycled the size 1 mean values instead of erroring, so rather than 2 rows, we end up with 5. df <- tibble( g = c(1, 1, 1, 2, 2), x = c(4, 3, 6, 2, 8), y = c(5, 1, 2, 8, 9) ) df #> # A tibble: 5 × 3 #> g x y #> #> 1 1 4 5 #> 2 1 3 1 #> 3 1 6 2 #> 4 2 2 8 #> 5 2 8 9 df |> summarise( x_average = mean(x), x_sd = x, # Oops .by = g ) #> Warning: Returning more (or less) than 1 row per `summarise()` group was deprecated in #> dplyr 1.1.0. #> ℹ Please use `reframe()` instead. #> ℹ When switching from `summarise()` to `reframe()`, remember that `reframe()` #> always returns an ungrouped data frame and adjust accordingly. #> # A tibble: 5 × 3 #> g x_average x_sd #> #> 1 1 4.33 4 #> 2 1 4.33 3 #> 3 1 4.33 6 #> 4 2 5 2 #> 5 2 5 8 [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) now throws a warning when any group returns a result that isn’t length 1. We expect to upgrade this to an error in the future to revert [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) back to its “safe” behavior of requiring 1 row per group. [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) also wasn’t the best name for a function with this feature, as the name itself implies one row per group. After [gathering some feedback](https://github.com/tidyverse/dplyr/issues/6565) , we’ve settled on a new verb with a more appropriate name, [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) . We think of [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) as a way to “do something” to each group, with no restrictions on the number of rows returned per group. The name has a nice connection to the tibble functions [`tibble::enframe()`](https://tibble.tidyverse.org/reference/enframe.html) and [`tibble::deframe()`](https://tibble.tidyverse.org/reference/enframe.html) , which are used for converting vectors to data frames and vice versa: * `enframe()`: Takes a vector, returns a data frame * `deframe()`: Takes a data frame, returns a vector * [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) : Takes a data frame, returns a data frame One nice application of [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) is computing quantiles at various probability thresholds. It’s particularly nice if we wrap [`quantile()`](https://rdrr.io/r/stats/quantile.html) into a helper that returns a data frame, which [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) then automatically unpacks. quantile_df <- function(x, probs = c(0.25, 0.5, 0.75)) { tibble( value = quantile(x, probs, na.rm = TRUE), prob = probs ) } df |> reframe(quantile_df(x), .by = g) #> # A tibble: 6 × 3 #> g value prob #> #> 1 1 3.5 0.25 #> 2 1 4 0.5 #> 3 1 5 0.75 #> 4 2 3.5 0.25 #> 5 2 5 0.5 #> 6 2 6.5 0.75 This also works well if you want to apply it to multiple columns using [`across()`](https://dplyr.tidyverse.org/reference/across.html) : df %>% reframe(across(x:y, quantile_df), .by = g) #> # A tibble: 6 × 3 #> g x$value $prob y$value $prob #> #> 1 1 3.5 0.25 1.5 0.25 #> 2 1 4 0.5 2 0.5 #> 3 1 5 0.75 3.5 0.75 #> 4 2 3.5 0.25 8.25 0.25 #> 5 2 5 0.5 8.5 0.5 #> 6 2 6.5 0.75 8.75 0.75 Because `quantile_df()` returns a tibble, we end up with [_packed_](https://tidyr.tidyverse.org/reference/pack.html) data frame columns. You’ll often want to unpack these into their individual columns, and [`across()`](https://dplyr.tidyverse.org/reference/across.html) has gained a new `.unpack` argument in 1.1.0 that helps you do exactly that: df %>% reframe(across(x:y, quantile_df, .unpack = TRUE), .by = g) #> # A tibble: 6 × 5 #> g x_value x_prob y_value y_prob #> #> 1 1 3.5 0.25 1.5 0.25 #> 2 1 4 0.5 2 0.5 #> 3 1 5 0.75 3.5 0.75 #> 4 2 3.5 0.25 8.25 0.25 #> 5 2 5 0.5 8.5 0.5 #> 6 2 6.5 0.75 8.75 0.75 We expect that seeing [`reframe()`](https://dplyr.tidyverse.org/reference/reframe.html) in a colleague’s code will serve as an extremely clear signal that something “special” is happening, because they’ve made a conscious decision to opt-into the 1% case of returning multiple rows per group. `arrange()`[](#arrange) ------------------------ We also mentioned in the [coming soon](https://www.tidyverse.org/blog/2022/11/dplyr-1-1-0-is-coming-soon/) post that [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) has undergone two user-facing changes: * When sorting character vectors, the C locale is now the default, rather than the system locale * A new `.locale` argument, powered by stringi, allows you to explicitly request an alternative locale using a stringi locale identifier (like `"en"` for English, or `"fr"` for French) These changes were made for two reasons: * Much faster performance by default, due to usage of a custom radix sort algorithm inspired by [data.table](https://cran.r-project.org/web/packages/data.table/index.html) ‘s `forder()` * Improved reproducibility across R sessions, where different computers might use different system locales and different operating systems have different ways to specify the same system locale If you use [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) for the purpose of grouping similar values together (and don’t care much about the specific locale that it uses to do so), then you’ll likely see performance improvements of up to 100x in dplyr 1.1.0. If you do care about the locale and supply `.locale`, you should still see improvements of up to 10x. # 10,000 random strings, sampled up to 1,000,000 rows dictionary <- stringi::stri_rand_strings(10000, length = 10, pattern = "[a-z]") str <- tibble(x = sample(dictionary, size = 1e6, replace = TRUE)) str #> # A tibble: 1,000,000 × 1 #> x #> #> 1 slpqkdtpyr #> 2 xtoucpndhc #> 3 vsvfoqcyqm #> 4 gnbpkwcmse #> 5 xutzdqxpsi #> 6 gkolsrndrz #> 7 mitqahkkou #> 8 eehfrrimhd #> 9 ymxxjczjsv #> 10 svpvizfxwe #> # … with 999,990 more rows # dplyr 1.0.10 (American English system locale) bench::mark(arrange(str, x)) #> # A tibble: 1 × 6 #> expression min median `itr/sec` mem_alloc `gc/sec` #> #> 1 arrange(str, x) 4.38s 4.89s 0.204 12.7MB 0.148 # dplyr 1.1.0 (C locale default, 100x faster) bench::mark(arrange(str, x)) #> # A tibble: 1 × 6 #> expression min median `itr/sec` mem_alloc `gc/sec` #> #> 1 arrange(str, x) 42.3ms 46.6ms 20.8 22.4MB 46.0 # dplyr 1.1.0 (American English `.locale`, 10x faster) bench::mark(arrange(str, x, .locale = "en")) #> # A tibble: 1 × 6 #> expression min median `itr/sec` mem_alloc #> #> 1 arrange(str, x, .locale = "en") 377ms 430ms 2.21 27.9MB #> # … with 1 more variable: `gc/sec` We are hopeful that switching to a C locale default will have a relatively small amount of impact in exchange for much faster performance. To read more about the exact differences between the C locale and locales like American English or Spanish, see the [coming soon](https://www.tidyverse.org/blog/2022/11/dplyr-1-1-0-is-coming-soon/#arrange-improvements-with-character-vectors) post or our detailed [tidyup](https://github.com/tidyverse/tidyups/blob/main/003-dplyr-radix-ordering.md) . If you are having trouble converting an existing script over to the new behavior, you can set the temporary global option `options(dplyr.legacy_locale = TRUE)`, which will revert to the pre-1.1.0 behavior of using the system locale. We expect to remove this option in a future release. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to the 88 contributors who helped make the 1.1.0 release possible by opening issues, contributing features and documentation, and asking questions! [@7708801314520dym](https://github.com/7708801314520dym) , [@abalter](https://github.com/abalter) , [@aghaynes](https://github.com/aghaynes) , [@AlbertRapp](https://github.com/AlbertRapp) , [@AlexGaithuma](https://github.com/AlexGaithuma) , [@algsat](https://github.com/algsat) , [@andrewbaxter439](https://github.com/andrewbaxter439) , [@andrewpbray](https://github.com/andrewpbray) , [@asadow](https://github.com/asadow) , [@asmlgkj](https://github.com/asmlgkj) , [@barbosawf](https://github.com/barbosawf) , [@barnabasharris](https://github.com/barnabasharris) , [@bart1](https://github.com/bart1) , [@bergsmat](https://github.com/bergsmat) , [@chrisbrownlie](https://github.com/chrisbrownlie) , [@cjyetman](https://github.com/cjyetman) , [@CNUlichao](https://github.com/CNUlichao) , [@daattali](https://github.com/daattali) , [@DanChaltiel](https://github.com/DanChaltiel) , [@davidchall](https://github.com/davidchall) , [@DavisVaughan](https://github.com/DavisVaughan) , [@ddsjoberg](https://github.com/ddsjoberg) , [@donboyd5](https://github.com/donboyd5) , [@drmowinckels](https://github.com/drmowinckels) , [@dxtxs1](https://github.com/dxtxs1) , [@eitsupi](https://github.com/eitsupi) , [@eogoodwin](https://github.com/eogoodwin) , [@erhoppe](https://github.com/erhoppe) , [@eutwt](https://github.com/eutwt) , [@ggrothendieck](https://github.com/ggrothendieck) , [@grayskripko](https://github.com/grayskripko) , [@H-Mateus](https://github.com/H-Mateus) , [@hadley](https://github.com/hadley) , [@haozhou1988](https://github.com/haozhou1988) , [@hassanjfry](https://github.com/hassanjfry) , [@Hesham999666](https://github.com/Hesham999666) , [@hideaki](https://github.com/hideaki) , [@jeffreypullin](https://github.com/jeffreypullin) , [@jic007](https://github.com/jic007) , [@jmbarbone](https://github.com/jmbarbone) , [@jonspring](https://github.com/jonspring) , [@jonthegeek](https://github.com/jonthegeek) , [@jpeacock29](https://github.com/jpeacock29) , [@kendonB](https://github.com/kendonB) , [@kenkoonwong](https://github.com/kenkoonwong) , [@kevinushey](https://github.com/kevinushey) , [@krlmlr](https://github.com/krlmlr) , [@larry77](https://github.com/larry77) , [@latot](https://github.com/latot) , [@lionel-](https://github.com/lionel-) , [@llayman12](https://github.com/llayman12) , [@LukasWallrich](https://github.com/LukasWallrich) , [@m-sostero](https://github.com/m-sostero) , [@machow](https://github.com/machow) , [@mc-unimi](https://github.com/mc-unimi) , [@mgacc0](https://github.com/mgacc0) , [@mgirlich](https://github.com/mgirlich) , [@MichelleSMA](https://github.com/MichelleSMA) , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel) , [@moodymudskipper](https://github.com/moodymudskipper) , [@moriarais](https://github.com/moriarais) , [@NicChr](https://github.com/NicChr) , [@nstjhp](https://github.com/nstjhp) , [@omarwh](https://github.com/omarwh) , [@orgadish](https://github.com/orgadish) , [@rempsyc](https://github.com/rempsyc) , [@rorynolan](https://github.com/rorynolan) , [@ryanvoyack](https://github.com/ryanvoyack) , [@selkamand](https://github.com/selkamand) , [@seth-cp](https://github.com/seth-cp) , [@shalom-lab](https://github.com/shalom-lab) , [@shannonpileggi](https://github.com/shannonpileggi) , [@simonpcouch](https://github.com/simonpcouch) , [@sjackson1997](https://github.com/sjackson1997) , [@spono](https://github.com/spono) , [@stibu81](https://github.com/stibu81) , [@tfehring](https://github.com/tfehring) , [@Theresaliu](https://github.com/Theresaliu) , [@TimBMK](https://github.com/TimBMK) , [@TimTeaFan](https://github.com/TimTeaFan) , [@Torvaney](https://github.com/Torvaney) , [@turbanisch](https://github.com/turbanisch) , [@weiyangtham](https://github.com/weiyangtham) , [@wurli](https://github.com/wurli) , [@xet869](https://github.com/xet869) , [@yuliaUU](https://github.com/yuliaUU) , [@yutannihilation](https://github.com/yutannihilation) , and [@zeehio](https://github.com/zeehio) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyr 1.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) tidyr 1.3.0 =========== ![](/blog/2023/01/tidyr-1-3-0/thumbnail-wd.jpg) Photo by [Jan Kopřiva](https://unsplash.com/photos/TEDo1eO8te4)   2023/01/24   [tidyr](/tags/tidyr/)   Hadley Wickham We’re pleased to announce the release of [tidyr](https://tidyr.tidyverse.org) 1.3.0. tidyr provides a set of tools for transforming data frames to and from tidy data, where each variable is a column and each observation is a row. Tidy data is a convention for matching the semantics and structure of your data that makes using the rest of the tidyverse (and many other R packages) much easier. You can install it from CRAN with: install.packages("tidyr") This post highlights the biggest changes in this release: * A new family of `separate_*()` functions supersede [`separate()`](https://tidyr.tidyverse.org/reference/separate.html) and [`extract()`](https://tidyr.tidyverse.org/reference/extract.html) and come with useful debugging features. * [`unnest_wider()`](https://tidyr.tidyverse.org/reference/unnest_wider.html) and [`unnest_longer()`](https://tidyr.tidyverse.org/reference/unnest_longer.html) gain a bundle of useful improvements. * [`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html) gets a new `cols_vary` argument. * `nest(.by)` provides a new (and hopefully final) way to create nested datasets. You should also notice generally improved errors with this release: we check function arguments more aggressively, and take care to always report the name of the function that you called, not some internal helper. As usual, you can find a full set of changes in the [release notes](http://github.com/tidyverse/tidyr/releases/tag/v1.3.0) . library(tidyr) library(dplyr, warn.conflicts = FALSE) `separate_*()` family of functions[](#separate_-family-of-functions) --------------------------------------------------------------------- The biggest feature of this release is a new, experimental family of functions for separating string columns: | | Make columns | Make rows | | --- | --- | --- | | Separate with delimiter | [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) | [`separate_longer_delim()`](https://tidyr.tidyverse.org/reference/separate_longer_delim.html) | | Separate by position | [`separate_wider_position()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) | [`separate_longer_position()`](https://tidyr.tidyverse.org/reference/separate_longer_delim.html) | | Separate with regular expression | [`separate_wider_regex()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) | | These functions collectively supersede [`extract()`](https://tidyr.tidyverse.org/reference/extract.html) , [`separate()`](https://tidyr.tidyverse.org/reference/separate.html) , and [`separate_rows()`](https://tidyr.tidyverse.org/reference/separate_rows.html) because they have more consistent names and arguments, have better performance (thanks to stringr), and provide a new approach for handling problems. | | Make columns | Make rows | | --- | --- | --- | | Separate with delimiter | `separate(sep = string)` | [`separate_rows()`](https://tidyr.tidyverse.org/reference/separate_rows.html) | | Separate by position | `separate(sep = integer vector)` | N/A | | Separate with regular expression | [`extract()`](https://tidyr.tidyverse.org/reference/extract.html) | | Here I’ll focus on the `wider` functions because they generally present the most interesting challenges. Let’s start by grabbing some census data with the [tidycensus](https://walker-data.com/tidycensus/) package: vt_census <- tidycensus::get_decennial( geography = "block", state = "VT", county = "Washington", variables = "P1_001N", year = 2020 ) #> Getting data from the 2020 decennial Census #> Using the PL 94-171 Redistricting Data summary file #> Note: 2020 decennial Census data use differential privacy, a technique that #> introduces errors into data to preserve respondent confidentiality. #> ℹ Small counts should be interpreted with caution. #> ℹ See https://www.census.gov/library/fact-sheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.html for additional guidance. #> This message is displayed once per session. vt_census #> # A tibble: 2,150 × 4 #> GEOID NAME variable value #> #> 1 500239555021014 Block 1014, Block Group 1, Census Tract 9555.… P1_001N 21 #> 2 500239555021015 Block 1015, Block Group 1, Census Tract 9555.… P1_001N 19 #> 3 500239555021016 Block 1016, Block Group 1, Census Tract 9555.… P1_001N 0 #> 4 500239555021017 Block 1017, Block Group 1, Census Tract 9555.… P1_001N 0 #> 5 500239555021018 Block 1018, Block Group 1, Census Tract 9555.… P1_001N 43 #> 6 500239555021019 Block 1019, Block Group 1, Census Tract 9555.… P1_001N 68 #> 7 500239555021020 Block 1020, Block Group 1, Census Tract 9555.… P1_001N 30 #> 8 500239555021021 Block 1021, Block Group 1, Census Tract 9555.… P1_001N 0 #> 9 500239555021022 Block 1022, Block Group 1, Census Tract 9555.… P1_001N 18 #> 10 500239555021023 Block 1023, Block Group 1, Census Tract 9555.… P1_001N 93 #> # … with 2,140 more rows The `GEOID` column is made up of four components: a 2-digit state identifier, a 3-digit county identifier, a 6-digit tract identifier, and a 4-digit block identifier. We can use [`separate_wider_position()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) to extract these into their own variables: vt_census |> select(GEOID) |> separate_wider_position( GEOID, widths = c(state = 2, county = 3, tract = 6, block = 4) ) #> # A tibble: 2,150 × 4 #> state county tract block #> #> 1 50 023 955502 1014 #> 2 50 023 955502 1015 #> 3 50 023 955502 1016 #> 4 50 023 955502 1017 #> 5 50 023 955502 1018 #> 6 50 023 955502 1019 #> 7 50 023 955502 1020 #> 8 50 023 955502 1021 #> 9 50 023 955502 1022 #> 10 50 023 955502 1023 #> # … with 2,140 more rows The `name` column contains this same information in a text form, with each component separated by a comma. We can use [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) to break up this sort of data into individual variables: vt_census |> select(NAME) |> separate_wider_delim( NAME, delim = ", ", names = c("block", "block_group", "tract", "county", "state") ) #> # A tibble: 2,150 × 5 #> block block_group tract county state #> #> 1 Block 1014 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 2 Block 1015 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 3 Block 1016 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 4 Block 1017 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 5 Block 1018 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 6 Block 1019 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 7 Block 1020 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 8 Block 1021 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 9 Block 1022 Block Group 1 Census Tract 9555.02 Washington County Vermont #> 10 Block 1023 Block Group 1 Census Tract 9555.02 Washington County Vermont #> # … with 2,140 more rows You’ll notice that each row contains a lot of duplicated information (“Block”, “Block Group”, …). You could certainly use [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) and string manipulation to clean this up, but there’s a more direct approach that you can use if you’re familiar with regular expressions. The new [`separate_wider_regex()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) takes a vector of regular expressions that are matched in order, from left to right. If you name the regular expression, it will appear in the output; otherwise, it will be dropped. I think this leads to a particularly elegant solution to many problems. vt_census |> select(NAME) |> separate_wider_regex( NAME, patterns = c( "Block ", block = "\\d+", ", ", "Block Group ", block_group = "\\d+", ", ", "Census Tract ", tract = "\\d+.\\d+", ", ", county = "[^,]+", ", ", state = ".*" ) ) #> # A tibble: 2,150 × 5 #> block block_group tract county state #> #> 1 1014 1 9555.02 Washington County Vermont #> 2 1015 1 9555.02 Washington County Vermont #> 3 1016 1 9555.02 Washington County Vermont #> 4 1017 1 9555.02 Washington County Vermont #> 5 1018 1 9555.02 Washington County Vermont #> 6 1019 1 9555.02 Washington County Vermont #> 7 1020 1 9555.02 Washington County Vermont #> 8 1021 1 9555.02 Washington County Vermont #> 9 1022 1 9555.02 Washington County Vermont #> 10 1023 1 9555.02 Washington County Vermont #> # … with 2,140 more rows These functions also have a new way to report problems. Let’s start with a very simple example: df <- tibble( id = 1:3, x = c("a", "a-b", "a-b-c") ) df |> separate_wider_delim(x, delim = "-", names = c("x", "y")) #> Error in `separate_wider_delim()`: #> ! Expected 2 pieces in each element of `x`. #> ! 1 value was too short. #> ℹ Use `too_few = "debug"` to diagnose the problem. #> ℹ Use `too_few = "align_start"/"align_end"` to silence this message. #> ! 1 value was too long. #> ℹ Use `too_many = "debug"` to diagnose the problem. #> ℹ Use `too_many = "drop"/"merge"` to silence this message. We’ve requested two columns in the output (`x` and `y`), but the first row has only one element and the last row has three elements, so [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) can’t do what we’ve asked. The error lays out your options for resolving the problem using the `too_few` and `too_many` arguments. I’d recommend always starting with `"debug"` to get more information about the problem: probs <- df |> separate_wider_delim( x, delim = "-", names = c("a", "b"), too_few = "debug", too_many = "debug" ) #> Warning: Debug mode activated: adding variables `x_ok`, `x_pieces`, and `x_remainder`. probs #> # A tibble: 3 × 7 #> id a b x x_ok x_pieces x_remainder #> #> 1 1 a NA a FALSE 1 "" #> 2 2 a b a-b TRUE 2 "" #> 3 3 a b a-b-c FALSE 3 "-c" This adds three new variables: `x_ok` tells you if the `x` could be separated as you requested, `x_pieces` tells you the actual number of pieces, and `x_remainder` shows you anything that remains after the columns you asked for. You can use this information to fix the problems in the input, or you can use the other options to `too_few` and `too_many` to tell [`separate_wider_delim()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) to fix them for you: df |> separate_wider_delim( x, delim = "-", names = c("a", "b"), too_few = "align_start", too_many = "drop" ) #> # A tibble: 3 × 3 #> id a b #> #> 1 1 a NA #> 2 2 a b #> 3 3 a b `too_few` and `too_many` also work with [`separate_wider_position()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) , and `too_few` works with [`separate_wider_regex()`](https://tidyr.tidyverse.org/reference/separate_wider_delim.html) . The `longer` variants don’t need these arguments because varying numbers of rows don’t matter in the same way that varying numbers of columns do: df |> separate_longer_delim(x, delim = "-") #> # A tibble: 6 × 2 #> id x #> #> 1 1 a #> 2 2 a #> 3 2 b #> 4 3 a #> 5 3 b #> 6 3 c These functions are still experimental so we are actively seeking feedback. Please try them out and let us know if you find them useful or if there are other features you’d like to see. `unnest_wider()` and `unnest_longer()` improvements[](#unnest_wider-and-unnest_longer-improvements) ---------------------------------------------------------------------------------------------------- [`unnest_longer()`](https://tidyr.tidyverse.org/reference/unnest_longer.html) and [`unnest_wider()`](https://tidyr.tidyverse.org/reference/unnest_wider.html) have both received some quality of life and consistency improvements. Most importantly: * [`unnest_wider()`](https://tidyr.tidyverse.org/reference/unnest_wider.html) now gives a better error when unnesting an unnamed vector: df <- tibble( id = 1:2, x = list(c("a", "b"), c("d", "e", "f")) ) df |> unnest_wider(x) #> Error in `unnest_wider()`: #> ℹ In column: `x`. #> ℹ In row: 1. #> Caused by error: #> ! Can't unnest elements with missing names. #> ℹ Supply `names_sep` to generate automatic names. df |> unnest_wider(x, names_sep = "_") #> # A tibble: 2 × 4 #> id x_1 x_2 x_3 #> #> 1 1 a b NA #> 2 2 d e f And this same behaviour now also applies to partially named vectors. * [`unnest_longer()`](https://tidyr.tidyverse.org/reference/unnest_longer.html) has gained a `keep_empty` argument like [`unnest()`](https://tidyr.tidyverse.org/reference/unnest.html) , and it now treats `NULL` and empty vectors the same way: df <- tibble( id = 1:3, x = list(NULL, integer(), 1:3) ) df |> unnest_longer(x) #> # A tibble: 3 × 2 #> id x #> #> 1 3 1 #> 2 3 2 #> 3 3 3 df |> unnest_longer(x, keep_empty = TRUE) #> # A tibble: 5 × 2 #> id x #> #> 1 1 NA #> 2 2 NA #> 3 3 1 #> 4 3 2 #> 5 3 3 `pivot_longer(cols_vary)`[](#pivot_longercols_vary) ---------------------------------------------------- By default, [`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html) creates its output row-by-row: df <- tibble( x = 1:2, y = 3:4, z = 5:6 ) df |> pivot_longer( everything(), names_to = "name", values_to = "value" ) #> # A tibble: 6 × 2 #> name value #> #> 1 x 1 #> 2 y 3 #> 3 z 5 #> 4 x 2 #> 5 y 4 #> 6 z 6 You can now request to create the output column-by-column with `cols_vary = "slowest":` df |> pivot_longer( everything(), names_to = "name", values_to = "value", cols_vary = "slowest" ) #> # A tibble: 6 × 2 #> name value #> #> 1 x 1 #> 2 x 2 #> 3 y 3 #> 4 y 4 #> 5 z 5 #> 6 z 6 `nest(.by)`[](#nestby) ----------------------- A nested data frame is a data frame where one (or more) columns is a list of data frames. Nested data frames are a powerful tool that allow you to turn groups into rows and can facilitate certain types of data manipulation that would be very tricky otherwise. (One place to learn more about them is my 2016 talk “ [Managing many models with R](https://www.youtube.com/watch?v=rz3_FDVt9eg) ".) Over the years we’ve made a number of attempts at getting the correct interface for nesting, including [`tidyr::nest()`](https://tidyr.tidyverse.org/reference/nest.html) , [`dplyr::nest_by()`](https://dplyr.tidyverse.org/reference/nest_by.html) , and [`dplyr::group_nest()`](https://dplyr.tidyverse.org/reference/group_nest.html) . In this version of tidyr we’ve taken one more stab at it by adding a new argument to [`nest()`](https://tidyr.tidyverse.org/reference/nest.html) : `.by`, inspired by the upcoming [dplyr 1.1.0](https://www.tidyverse.org/blog/2022/11/dplyr-1-1-0-is-coming-soon/) release. This means that [`nest()`](https://tidyr.tidyverse.org/reference/nest.html) now allows you to specify the variables you want to nest by as an alternative to specifying the variables that appear in the nested data. # Specify what to nest by mtcars |> nest(.by = cyl) #> # A tibble: 3 × 2 #> cyl data #> #> 1 6 #> 2 4 #> 3 8 # Specify what should be nested mtcars |> nest(data = -cyl) #> # A tibble: 3 × 2 #> cyl data #> #> 1 6 #> 2 4 #> 3 8 # Specify both (to drop variables) mtcars |> nest(data = mpg:drat, .by = cyl) #> # A tibble: 3 × 2 #> cyl data #> #> 1 6 #> 2 4 #> 3 8 If this function is all we hope it to be, we’re likely to supersede [`dplyr::nest_by()`](https://dplyr.tidyverse.org/reference/nest_by.html) and [`dplyr::group_nest()`](https://dplyr.tidyverse.org/reference/group_nest.html) in the future. This has the nice property of placing the functions for nesting and unnesting in the same package (tidyr). Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 51 contributors who helped make this release possible, by writing code and documentating, asking questions, and reporting bugs! [@AdrianS85](https://github.com/AdrianS85) , [@ahcyip](https://github.com/ahcyip) , [@allenbaron](https://github.com/allenbaron) , [@AnBarbosaBr](https://github.com/AnBarbosaBr) , [@ArthurAndrews](https://github.com/ArthurAndrews) , [@bart1](https://github.com/bart1) , [@billdenney](https://github.com/billdenney) , [@bknakker](https://github.com/bknakker) , [@bwiernik](https://github.com/bwiernik) , [@crissthiandi](https://github.com/crissthiandi) , [@daattali](https://github.com/daattali) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dcaud](https://github.com/dcaud) , [@DSLituiev](https://github.com/DSLituiev) , [@elgabbas](https://github.com/elgabbas) , [@fabiangehring](https://github.com/fabiangehring) , [@hadley](https://github.com/hadley) , [@ilikegitlab](https://github.com/ilikegitlab) , [@jennybc](https://github.com/jennybc) , [@jic007](https://github.com/jic007) , [@Joao-O-Santos](https://github.com/Joao-O-Santos) , [@joeycouse](https://github.com/joeycouse) , [@jonspring](https://github.com/jonspring) , [@kevinushey](https://github.com/kevinushey) , [@krlmlr](https://github.com/krlmlr) , [@lionel-](https://github.com/lionel-) , [@lotard](https://github.com/lotard) , [@lschneiderbauer](https://github.com/lschneiderbauer) , [@lucylgao](https://github.com/lucylgao) , [@markfairbanks](https://github.com/markfairbanks) , [@martina-starc](https://github.com/martina-starc) , [@MatthieuStigler](https://github.com/MatthieuStigler) , [@mattnolan001](https://github.com/mattnolan001) , [@mattroumaya](https://github.com/mattroumaya) , [@mdkrause](https://github.com/mdkrause) , [@mgirlich](https://github.com/mgirlich) , [@millermc38](https://github.com/millermc38) , [@modche](https://github.com/modche) , [@moodymudskipper](https://github.com/moodymudskipper) , [@mspittler](https://github.com/mspittler) , [@olivroy](https://github.com/olivroy) , [@piokol23](https://github.com/piokol23) , [@ppreshant](https://github.com/ppreshant) , [@ramiromagno](https://github.com/ramiromagno) , [@Rengervn](https://github.com/Rengervn) , [@rjake](https://github.com/rjake) , [@roohitk](https://github.com/roohitk) , [@struckma](https://github.com/struckma) , [@tjmahr](https://github.com/tjmahr) , [@weirichs](https://github.com/weirichs) , and [@wurli](https://github.com/wurli) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dplyr 1.1.0: Per-operation grouping [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dplyr 1.1.0: Per-operation grouping =================================== ![](/blog/2023/02/dplyr-1-1-0-per-operation-grouping/thumbnail-wd.jpg) Photo by [Clem Onojeghuo](https://www.pexels.com/photo/fruit-stand-375897/)   2023/02/01   [dplyr](/tags/dplyr/) , [dplyr-1-1-0](/tags/dplyr-1-1-0/)   Davis Vaughan Today we are going to look at one of the major new features in [dplyr 1.1.0](https://dplyr.tidyverse.org/news/index.html#dplyr-110) , per-operation grouping with [`.by`/`by`](https://dplyr.tidyverse.org/reference/dplyr_by.html) . Per-operation grouping is an experimental alternative to [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) which is only active within a single dplyr verb. This is another of the new dplyr features that was inspired by [data.table](https://cran.r-project.org/web/packages/data.table/index.html) , this time by their own grouping syntax with `by`. To see the other blog posts in this series, head [here](https://www.tidyverse.org/tags/dplyr-1-1-0/) . You can install it from CRAN with: install.packages("dplyr") library(dplyr) Persistent grouping with `group_by()`[](#persistent-grouping-with-group_by) ---------------------------------------------------------------------------- In dplyr, grouping radically affects the computation of the verb that you use it with. Since the very beginning of dplyr, you’ve been able to perform grouped operations by modifying your data frame with [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) . This grouping is _persistent_, meaning that it typically sticks around in some form for more than one operation. As an example, take a look at this `transactions` dataset which tracks revenue brought in from various transactions across multiple companies. If we wanted to add a column for the total yearly revenue per company, we might do: transactions <- tibble( company = c("A", "A", "A", "B", "B", "B"), year = c(2019, 2019, 2020, 2021, 2023, 2023), revenue = c(20, 50, 4, 10, 12, 18) ) transactions #> # A tibble: 6 × 3 #> company year revenue #> #> 1 A 2019 20 #> 2 A 2019 50 #> 3 A 2020 4 #> 4 B 2021 10 #> 5 B 2023 12 #> 6 B 2023 18 transactions |> group_by(company, year) |> mutate(total = sum(revenue)) #> # A tibble: 6 × 4 #> # Groups: company, year [4] #> company year revenue total #> #> 1 A 2019 20 70 #> 2 A 2019 50 70 #> 3 A 2020 4 4 #> 4 B 2021 10 10 #> 5 B 2023 12 30 #> 6 B 2023 18 30 Notice that the result is still grouped by both `company` and `year`. This is useful if you need to follow up with additional grouped operations (with the exact same grouping columns), but many people follow this [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) with an [`ungroup()`](https://dplyr.tidyverse.org/reference/group_by.html) . If we only need the totals, we could also use [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) , which peels off 1 layer of grouping by default: transactions |> group_by(company, year) |> summarise(total = sum(revenue)) #> `summarise()` has grouped output by 'company'. You can override using the #> `.groups` argument. #> # A tibble: 4 × 3 #> # Groups: company [2] #> company year total #> #> 1 A 2019 70 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 30 Here the grouping of the output isn’t exactly the same as the input, but we still consider this persistent grouping because some of the groups outlive the verb they were used with. Per-operation grouping with `.by`/`by`[](#per-operation-grouping-with-byby) ---------------------------------------------------------------------------- In dplyr 1.1.0, we’ve added an alternative to [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) known as [`.by`](https://dplyr.tidyverse.org/reference/dplyr_by.html) that introduces the idea of _per-operation_ grouping: transactions |> mutate(total = sum(revenue), .by = c(company, year)) #> # A tibble: 6 × 4 #> company year revenue total #> #> 1 A 2019 20 70 #> 2 A 2019 50 70 #> 3 A 2020 4 4 #> 4 B 2021 10 10 #> 5 B 2023 12 30 #> 6 B 2023 18 30 transactions |> summarise(total = sum(revenue), .by = c(company, year)) #> # A tibble: 4 × 3 #> company year total #> #> 1 A 2019 70 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 30 There are a few things about `.by` worth noting: * The result is always ungrouped, regardless of the number of grouping columns. With `.by`, you never need to remember to call [`ungroup()`](https://dplyr.tidyverse.org/reference/group_by.html) . * We used [tidyselect](https://tidyselect.r-lib.org/reference/language.html) to group by multiple columns. * [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) didn’t emit a message about regrouping. One of the things we like about `.by` is that it allows you to place the grouping specification alongside the code that uses it, rather than in a separate [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) line. This idea was inspired by data.table’s grouping syntax, which looks like: transactions[, .(total = sum(revenue)), by = .(company, year)] To see a complete list of dplyr verbs that support `.by`, look [here](https://dplyr.tidyverse.org/reference/dplyr_by.html#supported-verbs) . ### `.by` or `by`?[](#by-or-by) As you use per-operation grouping in dplyr, you’ll likely notice that some verbs use `.by` and others use `by`, for example: transactions |> slice_max(revenue, n = 2, by = company) #> # A tibble: 4 × 3 #> company year revenue #> #> 1 A 2019 50 #> 2 A 2019 20 #> 3 B 2023 18 #> 4 B 2023 12 This is a technical difference resulting from the fact that some verbs consistently use a `.` prefix for their arguments, and others don’t (see our design notes on the [dot prefix](https://design.tidyverse.org/dots-prefix.html) for more details). Most dplyr verbs use `.by`, and we’ve tried to ensure that the cases that are most likely to result in typos instead generate an informative error: # Uses `by` to be consistent with `n` and `prop` transactions |> slice_max(revenue, n = 2, .by = company) #> Error in `slice_max()`: #> ! Can't specify an argument named `.by` in this verb. #> ℹ Did you mean to use `by` instead? # Uses `.by` to be consistent with `.preserve` transactions |> slice(revenue, by = company) #> Error in `slice()`: #> ! Can't specify an argument named `by` in this verb. #> ℹ Did you mean to use `.by` instead? ### Translating from `group_by()`[](#translating-from-group_by) You shouldn’t feel pressured to translate existing code using [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) to use `.by` instead. [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) won’t ever disappear, and is not currently being superseded. That said, if you do want to start using `.by`, there are a few differences from [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) to be aware of. * `.by` always returns an ungrouped data frame. This is one of the main reasons to use `.by`, but is worth keeping in mind if you have existing code that takes advantage of persistent grouping from [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) . * `.by` uses tidy-selection. [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) , on the other hand, works more like [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) in that it allows you to create grouping columns on the fly, i.e. `df |> group_by(month = floor_date(date, "month"))`. With `.by`, you must create your grouping columns ahead of time. An added benefit of `.by`'s usage of tidy-selection is that you can supply an external character vector of grouping variables using `.by = all_of(groups_vec)`. * `.by` doesn’t sort grouping keys. [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) always sorts keys in ascending order, which affects the results of verbs like [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) . The last point might seem strange, but consider what happens if we preferred our transactions data in order by descending year so that the most recent transactions are at the top. transactions2 <- transactions |> arrange(company, desc(year)) transactions2 #> # A tibble: 6 × 3 #> company year revenue #> #> 1 A 2020 4 #> 2 A 2019 20 #> 3 A 2019 50 #> 4 B 2023 12 #> 5 B 2023 18 #> 6 B 2021 10 # Note that `group_by()` re-ordered transactions2 |> group_by(company, year) |> summarise(total = sum(revenue), .groups = "drop") #> # A tibble: 4 × 3 #> company year total #> #> 1 A 2019 70 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 30 # But `.by` used whatever order was already there transactions2 |> summarise(total = sum(revenue), .by = c(company, year)) #> # A tibble: 4 × 3 #> company year total #> #> 1 A 2020 4 #> 2 A 2019 70 #> 3 B 2023 30 #> 4 B 2021 10 Notice that `.by` doesn’t re-sort the grouping keys. Instead, the previous call to [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) is “respected” in the summary (this is also useful in combination with the new `.locale` argument to [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) ). We expect that most code won’t depend on the ordering of these group keys, but it is worth keeping in mind if you are switching to `.by`. If you did rely on sorted group keys, you currently need to explicitly call [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) either before or after the call to `summarise(.by =)`. In a future release, we may add [an argument](https://github.com/tidyverse/dplyr/issues/6663) to control this. `nest(.by = )`[](#nestby--) ---------------------------- library(tidyr) The idea behind `.by` turns out to be useful in contexts outside of dplyr. In [tidyr 1.3.0](https://www.tidyverse.org/blog/2023/01/tidyr-1-3-0/#nestby) , [`nest()`](https://tidyr.tidyverse.org/reference/nest.html) gained a `.by` argument, allowing you to specify the columns you want to nest _by_ rather than the columns that appear in the nested results, which often makes for more natural calls to [`nest()`](https://tidyr.tidyverse.org/reference/nest.html) . # Specify what to nest by transactions |> nest(.by = company) #> # A tibble: 2 × 2 #> company data #> #> 1 A #> 2 B # Specify what to nest transactions |> nest(data = !company) #> # A tibble: 2 × 2 #> company data #> #> 1 A #> 2 B # Specify both, allowing you to drop `year` along the way transactions |> nest(data = revenue, .by = company) #> # A tibble: 2 × 2 #> company data #> #> 1 A #> 2 B We currently have 3 different nesting variants in the tidyverse: [`tidyr::nest()`](https://tidyr.tidyverse.org/reference/nest.html) , [`dplyr::group_nest()`](https://dplyr.tidyverse.org/reference/group_nest.html) , and [`dplyr::nest_by()`](https://dplyr.tidyverse.org/reference/nest_by.html) . Because the tidyr variant is now the most flexible of all of these, and because [`unnest()`](https://tidyr.tidyverse.org/reference/unnest.html) also lives in tidyr, we are likely to deprecate the two experimental dplyr options in the future. Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyr [![](/blog/2023/01/tidyr-1-3-0/thumbnail-sq.jpg)](/blog/2023/01/tidyr-1-3-0/) [tidyr 1.3.0](/blog/2023/01/tidyr-1-3-0/) [package](/categories/package) Hadley Wickham tidyr 1.3.0 brings a new family of string separating functions, along with improvements to `unnest_longer()`, `unnest_wider()`, `pivot_longer()`, and `nest()`. [Read more ...](/blog/2023/01/tidyr-1-3-0/) 2023/01/24 [![](/blog/2022/02/tidyr-1-2-0/thumbnail-sq.jpg)](/blog/2022/02/tidyr-1-2-0/) [tidyr 1.2.0](/blog/2022/02/tidyr-1-2-0/) [package](/categories/package) Davis Vaughan tidyr 1.2.0 includes a bunch of new features and bug fixes, particularly for pivoting, rectangling, and grid specific tools. [Read more ...](/blog/2022/02/tidyr-1-2-0/) 2022/02/01 [![](/blog/2020/05/tidyr-1.1.0/thumbnail-sq.jpg)](/blog/2020/05/tidyr-1.1.0/) [tidyr 1.1.0](/blog/2020/05/tidyr-1.1.0/) [package](/categories/package) Hadley Wickham tidyr 1.1.0 includes a bunch of quality of life improvements, particularly for pivoting and rectangling. [Read more ...](/blog/2020/05/tidyr-1.1.0/) 2020/05/26 [![](/blog/2019/09/tidyr-1-0-0/tidyr-1-0-0-sq.jpg)](/blog/2019/09/tidyr-1-0-0/) [tidyr 1.0.0](/blog/2019/09/tidyr-1-0-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2019/09/tidyr-1-0-0/) 2019/09/13 [![](/blog/2018/02/tidyr-0-8-0/tidyr-0-8-0-sq.jpg)](/blog/2018/02/tidyr-0-8-0/) [tidyr 0.8.0](/blog/2018/02/tidyr-0-8-0/) [package](/categories/package) Hadley Wickham [Read more ...](/blog/2018/02/tidyr-0-8-0/) 2018/02/13 [![](/blog/2017/09/erratum-tidyr-0.7.0/erratum-tidyr-0.7.0-sq.jpg)](/blog/2017/09/erratum-tidyr-0.7.0/) [Erratum tidyr 0.7.0](/blog/2017/09/erratum-tidyr-0.7.0/) [package](/categories/package) Lionel Henry We have updated tidyselect to revert a behaviour introduced in tidyr 0.7.0. [Read more ...](/blog/2017/09/erratum-tidyr-0.7.0/) 2017/09/04 [![](/blog/2017/08/tidyr-0.7.0/tidyr-0.7.0-sq.jpg)](/blog/2017/08/tidyr-0.7.0/) [tidyr 0.7.0](/blog/2017/08/tidyr-0.7.0/) [package](/categories/package) Lionel Henry The next installment of tidyr is finally on CRAN! This version brings tidy eval to a crucial component of the tidyverse workflow. [Read more ...](/blog/2017/08/tidyr-0.7.0/) 2017/08/17 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # dbplyr 2.3.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) dbplyr 2.3.0 ============ ![](/blog/2023/01/dbplyr-2-3-0/thumbnail-wd.jpg) Photo by [Viktor Talashuk](https://unsplash.com/photos/05HLFQu8bFw)   2023/01/16   [dbplyr](/tags/dbplyr/) , [dplyr](/tags/dplyr/)   Hadley Wickham We’re chuffed to announce the release of [dbplyr](http://dbplyr.tidyverse.org/) 2.3.0. dbplyr is a database backend for dplyr that allows you to use a remote database as if it was a collection of local data frames: you write ordinary dplyr code and dbplyr translates it to SQL for you. You can install it from CRAN with: install.packages("{package}") This post will highlight some of the most important new features in 2.3.0: eliminating subqueries for many verb combinations, better errors, and a handful of new translations. As usual, this release comes with a large number of improvements to translations for individual backends and you can see the full list in the [release notes](%7B%20github_release%20%7D) library(dbplyr) library(dplyr, warn.conflicts = FALSE) SQL optimisation[](#sql-optimisation) -------------------------------------- dbplyr now produces fewer subqueries resulting in shorter, more readable, and, in some cases, faster SQL. The following combinations of verbs no longer require subqueries: * `*_join()` + [`select()`](https://dplyr.tidyverse.org/reference/select.html) and [`select()`](https://dplyr.tidyverse.org/reference/select.html) + `*_join()`. * [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) + [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) and [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) + [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) . * [`select()`](https://dplyr.tidyverse.org/reference/select.html) / [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) / [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) + [`distinct()`](https://dplyr.tidyverse.org/reference/distinct.html) . * [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) + [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) now translates to `HAVING`. * `left/inner_join()` + `left/inner_join()`. Here are a couple of examples of queries that are now much more compact: lf1 <- lazy_frame(x = 1, a = "a", .name = "lf1") lf2 <- lazy_frame(x = 1, b = "b", .name = "lf2") lf3 <- lazy_frame(x = 1, c = "c", .name = "lf3") lf1 |> left_join(lf2, by = "x") |> left_join(lf3, by = "x") |> select(b, c) #> #> SELECT `b`, `c` #> FROM `lf1` #> LEFT JOIN `lf2` #> ON (`lf1`.`x` = `lf2`.`x`) #> LEFT JOIN `lf3` #> ON (`lf1`.`x` = `lf3`.`x`) lf1 |> group_by(x) |> summarise(a = mean(a, na.rm = TRUE), n = n()) |> filter(n > 5) #> #> SELECT `x`, AVG(`a`) AS `a`, COUNT(*) AS `n` #> FROM `lf1` #> GROUP BY `x` #> HAVING (COUNT(*) > 5.0) (As ususal in these blog posts, I’m using [`lazy_frame()`](https://dbplyr.tidyverse.org/reference/tbl_lazy.html) to focus on the SQL generation, without having to set up a dummy database.) Additionally, where possible, dbplyr now uses `SELECT *` after a join instead of explicitly selecting every column. Improved errors[](#improved-errors) ------------------------------------ Variables that aren’t found in either the data or in the environment now produce an error: lf <- lazy_frame(x = 1,y = 2) lf |> mutate(x = z + 1) #> Error in `mutate()`: #> ! Problem while computing `x = z + 1` #> Caused by error: #> ! Object `z` not found. (Previously they were silently translated to SQL variables.) We’ve also generally reviewed the error messages to ensure they show more clearly where the error happened: lf |> mutate(x = y %/% 1) #> Error in `purrr::pmap()` at dbplyr/R/lazy-select-query.R:282:2: #> ℹ In index: 1. #> ℹ With name: x. #> Caused by error in `y %/% 1`: #> ! %/% is not available in this SQL variant lf |> mutate(across(x:y, "a")) #> Error in `mutate()`: #> ! Problem while computing `..1 = across(x:y, "a")` #> Caused by error in `across()`: #> ! `.fns` must be a NULL, a function, formula, or list New translations[](#new-translations) -------------------------------------- [`stringr::str_like()`](https://stringr.tidyverse.org/reference/str_like.html) (new in stringr 1.5.0) is translated to `LIKE`: lf1 |> filter(stringr::str_like(a, "abc")) #> #> SELECT * #> FROM `lf1` #> WHERE (`a` LIKE 'abc') dbplyr 2.3.0 is also supports features coming in [dplyr 1.1.0](https://www.tidyverse.org/blog/2022/11/dplyr-1-1-0-is-coming-soon/) : * The `.by` argument is supported as alternative to [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) . * Passing `...` to [`across()`](https://dplyr.tidyverse.org/reference/across.html) is deprecated because the evaluation timing of `...` is ambiguous. * New `pick()` and `case_match()` functions are translated. * [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) now supports the `.default` argument. This version does not support the new `join_by()` syntax, but we’re working on it, and we’ll release an update after dplyr 1.1.0 is out. Acknowledgements[](#acknowledgements) -------------------------------------- The vast majority of this release (particularly the SQL optimisations) are from [Maximilian Girlich](https://github.com/mgirlich) ; thanks so much for your continued work on this package. We’d also like to thank all 74 contributors who help in someway, whether it was filing issues or contributing code and documentation: [@a4sberg](https://github.com/a4sberg) , [@ablack3](https://github.com/ablack3) , [@akgold](https://github.com/akgold) , [@aleighbrown](https://github.com/aleighbrown) , [@andreassoteriadesmoj](https://github.com/andreassoteriadesmoj) , [@apalacio9502](https://github.com/apalacio9502) , [@baileych](https://github.com/baileych) , [@barnesparker](https://github.com/barnesparker) , [@bhuvanesh1707](https://github.com/bhuvanesh1707) , [@bkraft4257](https://github.com/bkraft4257) , [@bobbymc0](https://github.com/bobbymc0) , [@brian-law-rstudio](https://github.com/brian-law-rstudio) , [@bthe](https://github.com/bthe) , [@But2ene](https://github.com/But2ene) , [@capitantyler](https://github.com/capitantyler) , [@carlganz](https://github.com/carlganz) , [@cboettig](https://github.com/cboettig) , [@chwpearse](https://github.com/chwpearse) , [@copernican](https://github.com/copernican) , [@DSLituiev](https://github.com/DSLituiev) , [@ehudtr7](https://github.com/ehudtr7) , [@eitsupi](https://github.com/eitsupi) , [@ejneer](https://github.com/ejneer) , [@eutwt](https://github.com/eutwt) , [@ewright-vcan](https://github.com/ewright-vcan) , [@fabkury](https://github.com/fabkury) , [@fh-afrachioni](https://github.com/fh-afrachioni) , [@fh-mthomson](https://github.com/fh-mthomson) , [@filipemsc](https://github.com/filipemsc) , [@gadenbuie](https://github.com/gadenbuie) , [@gbouzill](https://github.com/gbouzill) , [@giocomai](https://github.com/giocomai) , [@hadley](https://github.com/hadley) , [@hershelm](https://github.com/hershelm) , [@iangow](https://github.com/iangow) , [@iMissile](https://github.com/iMissile) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@j-wester](https://github.com/j-wester) , [@Janlow](https://github.com/Janlow) , [@jasonmhoule](https://github.com/jasonmhoule) , [@jensmassberg](https://github.com/jensmassberg) , [@jmbarbone](https://github.com/jmbarbone) , [@joe-rodd](https://github.com/joe-rodd) , [@kongdd](https://github.com/kongdd) , [@krlmlr](https://github.com/krlmlr) , [@lschneiderbauer](https://github.com/lschneiderbauer) , [@machow](https://github.com/machow) , [@mgarbuzov](https://github.com/mgarbuzov) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@moodymudskipper](https://github.com/moodymudskipper) , [@multimeric](https://github.com/multimeric) , [@namarkus](https://github.com/namarkus) , [@noamross](https://github.com/noamross) , [@NZambranoc](https://github.com/NZambranoc) , [@oriolarques](https://github.com/oriolarques) , [@overmar](https://github.com/overmar) , [@owenjonesuob](https://github.com/owenjonesuob) , [@p-schaefer](https://github.com/p-schaefer) , [@rohitg33](https://github.com/rohitg33) , [@rowrowrowyourboat](https://github.com/rowrowrowyourboat) , [@rsund](https://github.com/rsund) , [@samssann](https://github.com/samssann) , [@samterfa](https://github.com/samterfa) , [@schradj](https://github.com/schradj) , [@scvail195](https://github.com/scvail195) , [@slhck](https://github.com/slhck) , [@splaisan](https://github.com/splaisan) , [@stephenashton-dhsc](https://github.com/stephenashton-dhsc) , [@ThomasMorland](https://github.com/ThomasMorland) , [@thothal](https://github.com/thothal) , [@viswaduttp](https://github.com/viswaduttp) , [@XoliloX](https://github.com/XoliloX) , and [@yuhenghuang](https://github.com/yuhenghuang) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # forcats 1.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) forcats 1.0.0 ============= ![](/blog/2023/01/forcats-1-0-0/thumbnail-wd.jpg) Photo by [Diego Morales](https://unsplash.com/photos/NWwv0ETyzxc)   2023/01/30   [forcats](/tags/forcats/)   Hadley Wickham We’re so happy to announce the release of [forcats](https://forcats.tidyverse.org) 1.0.0. The goal of the forcats package is to provide a suite of tools that solve common problems with factors, including changing the order of levels or the values. You can install it from CRAN with: install.packages("forcats") While this is the 1.0.0 release of forcats, this version number is mainly to signal that we think forcats is stable, and that we don’t anticipate any major changes in the future. This blog post will outline the only major new feature in this version: better tools for dealing with the two ways that missing values can be represented in factors. As usual, you can see a full list of changes in the [release notes](https://github.com/tidyverse/forcats/releases/tag/v1.0.0) . library(forcats) `NA` in levels vs `NA` in values[](#na-in-levels-vs-na-in-values) ------------------------------------------------------------------ There are two ways to represent a missing value in a factor: * You can include it in the values of the factor; it does not appear in the levels and [`is.na()`](https://rdrr.io/r/base/NA.html) reports it as missing. This is how missing values are encoded by default: f1 <- factor(c("x", "y", NA, NA, "x")) levels(f1) #> [1] "x" "y" is.na(f1) #> [1] FALSE FALSE TRUE TRUE FALSE * You can include it in the levels of the factor, thus [`is.na()`](https://rdrr.io/r/base/NA.html) does not report it as missing. This requires a little more work to create, because, by default, [`factor()`](https://rdrr.io/r/base/factor.html) uses `exclude = NA`, meaning that missing values are not included in the levels. You can force `NA` to be included by setting `exclude = NULL`: f2 <- factor(c("x", "y", NA, NA, "x"), exclude = NULL) levels(f2) #> [1] "x" "y" NA is.na(f2) #> [1] FALSE FALSE FALSE FALSE FALSE You can see the difference a little more clearly by looking at the underlying integer values of the factor: as.integer(f1) #> [1] 1 2 NA NA 1 as.integer(f2) #> [1] 1 2 3 3 1 When the `NA` is stored in the levels, there’s no missing value in the underlying integer values, because the value of level 3 is `NA`. `NA`s in the values tend to be best for data analysis, because [`is.na()`](https://rdrr.io/r/base/NA.html) works as you’d expect. `NA`s in the levels are useful if you need to control where missing values are shown in a table or a plot. To make it easier to switch between these forms, forcats now comes [`fct_na_value_to_level()`](https://forcats.tidyverse.org/reference/fct_na_value_to_level.html) and [`fct_na_level_to_value()`](https://forcats.tidyverse.org/reference/fct_na_value_to_level.html) . Here’s a practical example of why it matters. In the plot below, I’ve attempted to use [`fct_infreq()`](https://forcats.tidyverse.org/reference/fct_inorder.html) to reorder the levels of the factor so that the highest frequency levels are at the top of the bar chart: library(ggplot2) library(dplyr, warn.conflicts = FALSE) ggplot(starwars, aes(y = fct_rev(fct_infreq(hair_color)))) + geom_bar() + labs(y = "Hair color") ![The bar chart of hair color, now ordered so that the least frequent colours come first and the most frequent colors come last. This makes it easy to see that the most common hair color is none (~35), followed by brown (~18), then black (~12). Surprisingly, NAs are at the top of the graph, even though there are ~5 NAs and other colors have smaller values.](figs/fct-infreq-hair-1.png) Unfortunately, however, because the `NA`s are stored in the values, [`fct_infreq()`](https://forcats.tidyverse.org/reference/fct_inorder.html) has no ability to affect them, so they appear in their default position, after all the other values (it might not be obvious that that they’re after the other values here, but remember in plots y values have their smallest values at the bottom and highest values at the top). We can make [`fct_infreq()`](https://forcats.tidyverse.org/reference/fct_inorder.html) do what we want by moving the `NA` from the values to the levels: ggplot(starwars, aes(y = fct_rev(fct_infreq(fct_na_value_to_level(hair_color))))) + geom_bar() + labs(y = "Hair color") ![The bar chart of hair color, now ordered so that NAs are ordered where you'd expect: in between white (4) and black (12).](figs/unnamed-chunk-5-1.png) That code is getting a little verbose so lets pull it out into a separate dplyr step and pull the factor transformation in to its own mini pipeline: starwars |> mutate( hair_color = hair_color |> fct_na_value_to_level() |> fct_infreq() |> fct_rev() ) |> ggplot(aes(y = hair_color)) + geom_bar() + labs(y = "Hair color") ![](figs/unnamed-chunk-6-1.png) This structure makes it easier to make other adjustments. For example, the code below uses a more informative label for the missing level and lumps together the colours with less than 2 observations. I’ve left the (Other) category as a bar at the end, but if I wanted to I could cause it to sort in frequency order by flipping the order of [`fct_infreq()`](https://forcats.tidyverse.org/reference/fct_inorder.html) and [`fct_lump_min()`](https://forcats.tidyverse.org/reference/fct_lump.html) . starwars |> mutate( hair_color = hair_color |> fct_na_value_to_level("(Unknown)") |> fct_infreq() |> fct_lump_min(2, other_level = "(Other)") |> fct_rev() ) |> ggplot(aes(y = hair_color)) + geom_bar() + labs(y = "Hair color") ![The bar chart of hair color, with NA hair colour now labelled as (Unknown) and the low frequency bars lumped into (Other).](figs/unnamed-chunk-7-1.png) Looking closely at what got lumped together made me realise that there’s an existing “Unknown” level that should probably be represented as a missing value. One way to fix that is with [`fct_na_level_to_value()`](https://forcats.tidyverse.org/reference/fct_na_value_to_level.html) : starwars |> mutate( hair_color = hair_color |> fct_na_level_to_value("Unknown") |> fct_na_value_to_level("(Unknown)") |> fct_infreq() |> fct_lump_min(2, other_level = "(Other)") |> fct_rev() ) |> ggplot(aes(y = hair_color)) + geom_bar() + labs(y = "Hair color") ![The bar chart of hair color, with "unknown" hair colour now lumped in with (Unknown) instead of other](figs/unnamed-chunk-8-1.png) Acknowledgements[](#acknowledgements) -------------------------------------- Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q4 2022 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q4 2022 tidymodels digest ========================= ![](/blog/2022/12/tidymodels-2022-q4/thumbnail-wd.jpg) Photo by [Ilona Frey](https://unsplash.com/photos/WKWENifHmJw)   2022/12/29   [tidymodels](/tags/tidymodels/) , [recipes](/tags/recipes/) , [parsnip](/tags/parsnip/) , [rsample](/tags/rsample/)   Simon Couch The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like these posts from the past couple months: * [tidyclust is on CRAN](https://www.tidyverse.org/blog/2022/12/tidyclust-0-1-0/) * [Model calibration](https://www.tidyverse.org/blog/2022/11/model-calibration/) * [Improvements to model specification checking in tidymodels](https://www.tidyverse.org/blog/2022/10/parsnip-checking-1-0-2/) Since [our last roundup post](https://www.tidyverse.org/blog/2022/10/tidymodels-2022-q3/) , there have been CRAN releases of 9 tidymodels packages. Here are links to their NEWS files: * bonsai [(0.2.1)](https://bonsai.tidymodels.org/news/index.html) * broom [(1.0.2)](https://broom.tidymodels.org/news/index.html) * butcher [(0.3.1)](https://butcher.tidymodels.org/news/index.html) * dials [(1.1.0)](https://dials.tidymodels.org/news/index.html) * parsnip [(1.0.3)](https://parsnip.tidymodels.org/news/index.html) * recipes [(1.0.3)](https://recipes.tidymodels.org/news/index.html) * rsample [(1.1.1)](https://rsample.tidymodels.org/news/index.html) * stacks [(1.0.1)](https://stacks.tidymodels.org/news/index.html) * workflows [(1.1.2)](https://workflows.tidymodels.org/news/index.html) We’ll highlight a few especially notable changes below: more specialized role selectors in recipes, extended support for grouped resampling in rsample, and a big speedup in parsnip. First, loading the collection of packages: library(tidymodels) Specialized role selectors[](#specialized-role-selectors) ---------------------------------------------------------- The [recipes package for preprocessing](https://recipes.tidymodels.org/) supports tidyselect-style variable selection, and includes some of its own selectors to support common modeling workflows. To illustrate, we’ll make use of a dataset `goofy_data` with a number of different variable types: str(goofy_data) #> tibble [100 × 10] (S3: tbl_df/tbl/data.frame) #> $ class: Factor w/ 2 levels "class_1","class_2": 1 1 2 1 2 1 1 2 2 2 ... #> $ a : Factor w/ 7 levels "-3","-2","-1",..: 4 4 3 2 4 5 2 2 3 5 ... #> $ b : Factor w/ 9 levels "-4","-3","-2",..: 9 5 4 3 4 7 4 2 3 6 ... #> $ c : int [1:100] 0 0 0 0 0 0 0 -1 0 1 ... #> $ d : int [1:100] 0 1 1 1 0 1 1 0 0 1 ... #> $ e : int [1:100] 1 0 1 0 0 1 1 0 1 1 ... #> $ f : num [1:100] 1.01 -1.99 2.18 2.3 -3.01 ... #> $ g : num [1:100] -0.845 1.456 1.948 1.354 1.085 ... #> $ h : num [1:100] -0.285 0.59 -0.938 1.447 0.424 ... #> $ i : chr [1:100] "white" "maroon" "maroon" "maroon" ... Imagine a classification problem on the `goofy_data` where we’d like to predict `class` using the remaining variables as predictors. The selector functions allow us to perform operations on only the predictors with a certain class. For instance, centering and scaling all numeric predictors: recipe(class ~ ., goofy_data) %>% step_normalize(all_numeric_predictors()) %>% prep() #> Recipe #> #> Inputs: #> #> role #variables #> outcome 1 #> predictor 9 #> #> Training data contained 100 data points and no missing data. #> #> Operations: #> #> Centering and scaling for c, d, e, f, g, h [trained] Or making dummy variables out of each of the nominal predictors: recipe(class ~ ., goofy_data) %>% step_dummy(all_nominal_predictors()) %>% prep() #> Recipe #> #> Inputs: #> #> role #variables #> outcome 1 #> predictor 9 #> #> Training data contained 100 data points and no missing data. #> #> Operations: #> #> Dummy variables from a, b, i [trained] Operations like those above have been long-standing functionality in recipes, and are powerful tools for effective modeling. The most recent release of recipes introduced [finer-grain selectors](https://fosstodon.org/@emilhvitfeldt/109315135944110742) for variable types. For instance, we may want to only center and scale the _double_ (i.e. real-valued) predictors, excluding the integers. With the new release of recipes, we can easily do so: recipe(class ~ ., goofy_data) %>% step_normalize(all_double_predictors()) %>% prep() #> Recipe #> #> Inputs: #> #> role #variables #> outcome 1 #> predictor 9 #> #> Training data contained 100 data points and no missing data. #> #> Operations: #> #> Centering and scaling for f, g, h [trained] This is one of a number of new selectors: * The `all_nominal()` selector now has finer-grained versions `all_string()`, `all_factor()`, `all_unordered()`, and `all_ordered()`. * The `all_numeric()` selector now has finer-grained versions `all_double()`, and `all_integer()`. * New `all_logical()`, `all_date()`, and `all_datetime()` selectors. All new selectors have `*_predictors()` variants. You can read more about recipes 1.0.3 in the [release notes](https://recipes.tidymodels.org/news/index.html#recipes-103) . Grouped resampling[](#grouped-resampling) ------------------------------------------ The most recent release of rsample introduced support for stratification with grouped resampling. Consider the following toy data set on the number of melons in a household: melons #> # A tibble: 4,928 × 3 #> household n_melons chops #> #> 1 1 114 Yes #> 2 1 179 Yes #> 3 1 163 Yes #> 4 1 35 Yes #> 5 1 93 Yes #> 6 1 55 Yes #> 7 1 165 Yes #> 8 1 30 Yes #> 9 1 140 Yes #> 10 1 7 Yes #> # … with 4,918 more rows There are 100 different households in this dataset. Each member of the household has some number of melons `n_melons` in their fridge. A household, i.e., all its members, either `chops` their melons or keeps them whole. Each of the resampling functions in rsample have a `group_*`ed analogue. From rsample’s [“Common Patterns” article](https://rsample.tidymodels.org/articles/Common_Patterns.html#grouped-resampling) : > Often, some observations in your data will be “more related” to each other than would be probable under random chance, for instance because they represent repeated measurements of the same subject or were all collected at a single location. In these situations, you often want to assign all related observations to either the analysis or assessment fold as a group, to avoid having assessment data that's closely related to the data used to fit a model. For example, the grouped `initial_split()` variant will allot the training and testing set mutually exclusive levels of the `group` variable: resample <- group_initial_split(melons, group = household) sum( unique(training(resample)$household) %in% unique(testing(resample)$household) ) #> [1] 0 However, note that there are only a few households that don’t chop their melons, and those households tend to have many more melons to chop! ![A ggplot histogram displaying the mean number of melons per household, filled by whether the household chops their melons or not. The plot shows that there are relatively few households that don't chop their melons, but those households have many more melons to chop. Households that chop their melons have around 80 to chop, while those that don't have around 200.](figs/melon-plot-1.png) If we’re ultimately interested in modeling whether a household chops their melons, we ought to ensure that both values of `chops` are well-represented in both the training and testing set. The argument `strata = chops` indicates that sampling by `household` will occur within values of `chops`. Note that the strata must be constant in each group, so here, all members of a household need to either chop or not. resample_stratified <- group_initial_split(melons, group = household, strata = chops) Note that this resampling scheme still resulted in different `household`s being allotted to training and testing: sum( unique(training(resample_stratified)$household) %in% unique(testing(resample_stratified)$household) ) #> [1] 0 Also, though, it ensured that similar proportions of `chops` values are allotted to the training and testing set: diff(c( mean(training(resample_stratified)$chops == "Yes"), mean(testing(resample_stratified)$chops == "Yes") )) #> [1] 0.01000042 You can read more about rsample 1.1.1 in the [release notes](https://rsample.tidymodels.org/news/index.html#rsample-111) . Performance speedup[](#performance-speedup) -------------------------------------------- We recently made a performance tweak, released as part of parsnip 1.0.3, that resulted in a substantial speedup in fit time. Fitting models via parsnip is a fundamental operation in the tidymodels, so the speedup can be observed across many modeling workflows. The figure below demonstrates this speedup in [an experiment](https://gist.github.com/simonpcouch/651d0ea4d968b455ded8194578dabf52) involving fitting a simple linear regression model on resamples of simulated data. Simulated datasets with between one hundred and one million rows were partitioned into five, ten, or twenty folds and fitted with the new version of parsnip as well as the version preceding it. With smaller datasets, the speedup is negligible, but fit times decrease by a factor of three to five once training data reaches one million rows. ![A ggplot line plot displaying the relative speedup between parsnip 1.0.2 and 1.0.3. The number of rows in training data is on the x axis, ranging from one hundred to one million, and the factor of speedup (1.0.2 over 1.0.3) is on the y axis, ranging from 1 to 5. Three lines, colored by 'number of folds,' noting 5, 10, or 20 resamples, stretch from the bottom left to top right of the plot. This shows that, as training data gets larger, the magnitude of speedup with the new parsnip version gets larger and larger.](figs/speedup-1.png) You can read more about parsnip 1.0.3 in the [release notes](https://parsnip.tidymodels.org/news/index.html#parsnip-103) . Acknowledgements[](#acknowledgements) -------------------------------------- We’d like to thank those in the community that contributed to tidymodels in the last quarter: * bonsai: [@HenrikBengtsson](https://github.com/HenrikBengtsson) , and [@simonpcouch](https://github.com/simonpcouch) . * broom: [@amorris28](https://github.com/amorris28) , [@capnrefsmmat](https://github.com/capnrefsmmat) , [@larmarange](https://github.com/larmarange) , [@lukepilling](https://github.com/lukepilling) , and [@simonpcouch](https://github.com/simonpcouch) . * butcher: [@galen-ft](https://github.com/galen-ft) , and [@juliasilge](https://github.com/juliasilge) . * dials: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , and [@Tadge-Analytics](https://github.com/Tadge-Analytics) . * parsnip: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@exsell-jc](https://github.com/exsell-jc) , [@fkohrt](https://github.com/fkohrt) , [@hfrick](https://github.com/hfrick) , [@jonthegeek](https://github.com/jonthegeek) , [@Marwolaeth](https://github.com/Marwolaeth) , [@mattwarkentin](https://github.com/mattwarkentin) , [@schoonees](https://github.com/schoonees) , [@simonpcouch](https://github.com/simonpcouch) , [@sweiner123](https://github.com/sweiner123) , and [@topepo](https://github.com/topepo) . * recipes: [@andeek](https://github.com/andeek) , [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@joeycouse](https://github.com/joeycouse) , [@mdancho84](https://github.com/mdancho84) , and [@mobius-eng](https://github.com/mobius-eng) . * rsample: [@bschneidr](https://github.com/bschneidr) , [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@mikemahoney218](https://github.com/mikemahoney218) , [@pgg1309](https://github.com/pgg1309) , and [@topepo](https://github.com/topepo) . * stacks: [@simonpcouch](https://github.com/simonpcouch) . * workflows: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@simonpcouch](https://github.com/simonpcouch) , [@talegari](https://github.com/talegari) , and [@xiaochi-liu](https://github.com/xiaochi-liu) . We’re grateful for all of the tidymodels community, from observers to users to contributors, and wish you all a happy new year! Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Model Calibration [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Model Calibration ================= ![](/blog/2022/11/model-calibration/thumbnail-wd.jpg) Photo by [Graphic Node](https://unsplash.com/photos/s3B_pjK7UIs)   2022/11/29   [model](/tags/model/) , [plots](/tags/plots/)   Edgar Ruiz I am very excited to introduce work currently underway on the probably package. We are looking to create early awareness and receive feedback from the community. That is why the enhancements discussed here are not yet on CRAN. While the article is meant to introduce new package functionality, we also have the goal of introducing model calibration conceptually. We want to provide sufficient background for those who may not be familiar with model calibration. If you are already familiar with this technique, feel free to skip to the [Setup](#example-data) section to get started. To install the version of probably used here: remotes::install_github("tidymodels/probably") Model Calibration[](#model-calibration) ---------------------------------------- _The goal of model calibration is to ensure that the estimated class probabilities are consistent with what would naturally occur._ If a model has poor calibration, we might be able to post-process the original predictions to coerce them to have better properties. There are two main components to model calibration: * **Diagnosis** - Figuring out how well the original (and re-calibrated) probabilities perform. * **Remediation** - Adjusting the original values to have better properties. ### The Development Plan[](#the-development-plan) As with everything in machine learning, there are several options to consider when calibrating a model. Through the new features in the tidymodels packages, we aspire to make those options as easily accessible as possible. Our plan is to implement model calibration in two phases: the first phase will focus on binary models, and the second phase will focus on multi-class models. The first batch of enhancements are now available in the development version of the probably package. The enhancements are centered around plotting functions meant for **diagnosing** the prediction’s performance. These are more commonly known as **calibration plots**. Calibration Plots[](#calibration-plots) ---------------------------------------- The idea behind a calibration plot is that if we group the predictions based on their probability, then we should see a percentage of events [1](#fn:1) that match such probability. For example, if we collect a group of the predictions whose probabilities are estimated to be about 10%, then we should expect that about 10% of the those in the group to indeed be events. The plots shown below can be used as diagnostics to see if our predictions are consistent with the observed event rates. ### Example Data[](#example-data) If you would like to follow along, load the probably and dplyr packages into your R session. library(tidymodels) library(probably) The probably package comes with a few data sets. For most of the examples in this post, we will use `segment_logistic`, an example data set that contains predicted probabilities and classes from a logistic regression model for a binary outcome `Class`, taking values `"good"` or `"bad"`. predictions, and their probabilities. `Class` contains the outcome of `.pred_good` contains the probability that the event is “good”. segment_logistic #> # A tibble: 1,010 × 3 #> .pred_poor .pred_good Class #> * #> 1 0.986 0.0142 poor #> 2 0.897 0.103 poor #> 3 0.118 0.882 good #> 4 0.102 0.898 good #> 5 0.991 0.00914 poor #> 6 0.633 0.367 good #> 7 0.770 0.230 good #> 8 0.00842 0.992 good #> 9 0.995 0.00458 poor #> 10 0.765 0.235 poor #> # … with 1,000 more rows ### Binned Plot[](#binned-plot) On smaller data sets, it is challenging to obtain an accurate _event rate_ for a given probability. For example, if there are 5 predictions with about a 50% probability, and 3 of those are events, the plot would show a 60% event rate. This comparison would not be appropriate because there are not enough predictions to determine how close to 50% the model really is. The most common approach is to group the probabilities into bins, or buckets. Usually, the data is split into 10 discrete buckets, from 0 to 1 (0 - 100%). The _event rate_ and the _bin midpoint_ is calculated for each bin. In the probably package, binned calibration plots can be created using [`cal_plot_breaks()`](https://probably.tidymodels.org/reference/cal_plot_breaks.html) . It expects a data set (`.data`), the un-quoted variable names that contain the events (`truth`), and the probabilities (`estimate`). For the example here, we pass the `segment_logistic` data set, and use `Class` and `.pred_good` as the arguments. By default, this function will create a calibration plot with 10 buckets (breaks): segment_logistic %>% cal_plot_breaks(Class, .pred_good) ![A ggplot line plot with predicted probabilities on the x axis and event rates on the y axis, both ranging from 0 to 1. A dashed line lies on the identity line y equals x, and is loosely followed by a solid line that joins a series of dots representing the midpoint for each of 10 bins. Past predicted probabilities of 0.5, the dots consistently lie below the dashed line.](figs/unnamed-chunk-4-1.png) The calibration plot for the ideal model will essentially be perfect incline line that start at (0,0) and ends in (1,1). In the case of this model, we can see that the seventh point has an event rate of 49.1% despite having estimated probabilities ranging from 60% to 70%. This indicates that the model is not creating predictions in this region that are consistent with the data (i.e., it is under-predicting). The number of bins in [`cal_plot_breaks()`](https://probably.tidymodels.org/reference/cal_plot_breaks.html) can be adjusted using `num_breaks`. Here is an example of what the plot looks like if we reduce the bins from 10, to 5: segment_logistic %>% cal_plot_breaks(Class, .pred_good, num_breaks = 5 ) ![A calibration like that above, but with half as many bins. In this version of the plot, the solid line is less jagged, though still shows that dots consistently lie below the dashed line beyond a predicted probability of 0.5.](figs/unnamed-chunk-6-1.png) The number of breaks should be based on ensuring that there is enough data in each bin to adequately estimate the observed event rate. If your data are small, the next version of the calibration plot might be a better solution. ### Windowed[](#windowed) Another approach is to use overlapping ranges, or windows. Like the previous plot, we bin the data and calculate the event rate. However, we can add more bins by allowing them to overlap. If the data set size is small, one strategy is to use a set of wide bins that overlap one another. There are two variables that control the windows. The **step size** controls the frequency of the windows. If we set a step size of 5%, windows will be created for each 5% increment in predicted probability (5%, 10%, 15%, etc). The second argument is the (maximum) **window size**. If it is set to 10%—and the step size is set at 5%—then a given step will overlap halfway into both the previous step and the next step. Here is a visual representation of this specific scenario: ![Plot illustrating the horizontal location of each step and the size of the window](figs/unnamed-chunk-7-1.png) In probably, the [`cal_plot_windowed()`](https://probably.tidymodels.org/reference/cal_plot_breaks.html) function provides this functionality. The default step size is 0.05, and can be changed via the `step_size` argument. The default window size is 0.1, and can be changed via the `window_size` argument: segment_logistic %>% cal_plot_windowed(Class, .pred_good) ![Calibration plot with 21 windows, created with the cal_plot_windowed() function](figs/unnamed-chunk-8-1.png) Here is an example of reducing the `step_size` from 0.05 to 0.02. There are more than double the windows: segment_logistic %>% cal_plot_windowed(Class, .pred_good, step_size = 0.02) ![Calibration plot with more steps than the default, created with the cal_plot_windowed() function](figs/unnamed-chunk-9-1.png) ### Model-Based[](#model-based) Another way to visualize the performance is to fit a classification model of the events against the estimated probabilities. This is helpful because it avoids the use of pre-determined groupings. Another difference is that we are not plotting midpoints of actual results, but rather predictions based on those results. The [`cal_plot_logistic()`](https://probably.tidymodels.org/reference/cal_plot_breaks.html) provides this functionality. By default, it uses a logistic regression. There are two possible methods for fitting: * `smooth = TRUE` (the default) fits a generalized additive model using splines. This allows for more flexible model fits. * `smooth = FALSE` uses an ordinary logistic regression model with linear terms for the predictor. As an example: segment_logistic %>% cal_plot_logistic(Class, .pred_good) ![Logistic Spline calibration plot, created with the cal_plot_logistic() function](figs/unnamed-chunk-10-1.png) The corresponding [`glm()`](https://rdrr.io/r/stats/glm.html) model produces: segment_logistic %>% cal_plot_logistic(Class, .pred_good, smooth = FALSE) ![Ordinary logistic calibration plot, created with the cal_plot_logistic() function](figs/unnamed-chunk-11-1.png) ### Additional options and features[](#additional-options-and-features) #### **Intervals**[](#intervals) The confidence intervals are visualized using the gray ribbon. The default interval is 0.9, but can be changed using the `conf_level` argument. segment_logistic %>% cal_plot_breaks(Class, .pred_good, conf_level = 0.8) ![Calibration plot with a confidence interval set to 0.8](figs/unnamed-chunk-12-1.png) If desired, the intervals can be removed by setting the `include_ribbon` argument to `FALSE`. segment_logistic %>% cal_plot_breaks(Class, .pred_good, include_ribbon = FALSE) ![Calibration plot with the confidence interval ribbon turned off](figs/unnamed-chunk-13-1.png) #### **Rugs**[](#rugs) By default, the calibration plots include a RUGs layer at the top and at the bottom of the visualization. They are meant to give us an idea of the density of events and non-events as the probabilities progress from 0 to 1. ![Calibration plot with arrows pointing to where the RUGS plots are placed in the graph](figs/unnamed-chunk-14-1.png) This layer can be removed by setting `include_rug` to `FALSE`: segment_logistic %>% cal_plot_breaks(Class, .pred_good, include_rug = FALSE) ![Calibration plot without RUGS](figs/unnamed-chunk-15-1.png) Integration with tune[](#integration-with-tune) ------------------------------------------------ So far, the inputs to the functions have been data frames. In tidymodels, the tune package has methods for resampling models as well as functions for tuning hyperparameters. The calibration plots in the probably package also support the results of these functions (with class `tune_results`). The functions read the metadata from the tune object, and the `truth` and `estimate` arguments automatically. To showcase this feature, we will tune a model based on simulated data. In order for the calibration plot to work, the predictions need to be collected. This is done by setting `save_pred` to `TRUE` in `tune_grid()`'s control settings. set.seed(111) sim_data <- sim_classification(500) sim_folds <- vfold_cv(sim_data, repeats = 3) rf_mod <- rand_forest(min_n = tune()) %>% set_mode("classification") set.seed(222) tuned_model <- rf_mod %>% tune_grid( class ~ ., resamples = sim_folds, grid = 4, # Important: `saved_pred` has to be set to TRUE in order for # the plotting to be possible control = control_resamples(save_pred = TRUE) ) tuned_model #> # Tuning results #> # 10-fold cross-validation repeated 3 times #> # A tibble: 30 × 6 #> splits id id2 .metrics .notes .predicti…¹ #> #> 1 Repeat1 Fold01 #> 2 Repeat1 Fold02 #> 3 Repeat1 Fold03 #> 4 Repeat1 Fold04 #> 5 Repeat1 Fold05 #> 6 Repeat1 Fold06 #> 7 Repeat1 Fold07 #> 8 Repeat1 Fold08 #> 9 Repeat1 Fold09 #> 10 Repeat1 Fold10 #> # … with 20 more rows, and abbreviated variable name ¹​.predictions The plotting functions will automatically collect the predictions. Each of the pre-processing groups will be plotted individually in its own facet. tuned_model %>% cal_plot_logistic() ![Multiple calibration plots presented in a grid](figs/unnamed-chunk-17-1.png) A panel is produced for each value of `min_n`, coded with an automatically generated configuration name. This makes sure to use the out-of-sample data to make the plot (instead of just re-predicting the training set). Preparing for the next stage[](#preparing-for-the-next-stage) -------------------------------------------------------------- As mentioned in the outset of this post, the goal is to also provide a way to calibrate the model, and to apply the calibration to future predictions. We have made sure that the plotting functions are ready now to accept multiple probability sets. In this post, we will showcase that functionality by “manually” creating a quick calibration model and comparing its output to the original probabilities. We will need both of them in the same data frame, as well as a variable distinguishing the original probabilities from the calibrated probabilities. In this case we will create a variable called `source`: model <- glm(Class ~ .pred_good, segment_logistic, family = "binomial") preds <- predict(model, segment_logistic, type = "response") combined <- bind_rows( mutate(segment_logistic, source = "original"), mutate(segment_logistic, .pred_good = 1 - preds, source = "glm") ) combined #> # A tibble: 2,020 × 4 #> .pred_poor .pred_good Class source #> #> 1 0.986 0.0142 poor original #> 2 0.897 0.103 poor original #> 3 0.118 0.882 good original #> 4 0.102 0.898 good original #> 5 0.991 0.00914 poor original #> 6 0.633 0.367 good original #> 7 0.770 0.230 good original #> 8 0.00842 0.992 good original #> 9 0.995 0.00458 poor original #> 10 0.765 0.235 poor original #> # … with 2,010 more rows The new plot functions support dplyr groupings. So, to overlay the two groups, we just need to pass `source` to [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) : combined %>% group_by(source) %>% cal_plot_breaks(Class, .pred_good) ![Calibration plot with two overlaying probability trends, one is the original and the second is the model](figs/unnamed-chunk-19-1.png) If we would like to plot them side by side, we can add [`facet_wrap()`](https://ggplot2.tidyverse.org/reference/facet_wrap.html) as an additional step of the plot: combined %>% group_by(source) %>% cal_plot_breaks(Class, .pred_good) + facet_wrap(~source) + theme(legend.position = "none") ![Calibration plot with two side-by-side probability trends](figs/unnamed-chunk-20-1.png) Our goal in the future is to provide calibration functions that create the models, and provide an easy way to visualize them. Conclusion[](#conclusion) -------------------------- As mentioned at the top of this post, we welcome your feedback as you try out these features and read about our plans for the future. If you wish to send us your thoughts, feel free to open an issue in probably’s GitHub repo here: [https://github.com/tidymodels/probably/issues](https://github.com/tidymodels/probably/issues) . * * * 1. We can think of an **event** as the outcome that is being tracked by the probability. For example, if a model predicts “heads” or “tails” and we want to calibrate the probability for “tails”, then the **event** is when the column containing the outcome, has the value of “tails”. [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # stringr [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Stringr [![](/blog/2022/12/stringr-1-5-0/thumbnail-sq.jpg)](/blog/2022/12/stringr-1-5-0/) [stringr 1.5.0](/blog/2022/12/stringr-1-5-0/) [package](/categories/package) Hadley Wickham It’s been a long three years but a new version of stringr is now on CRAN! This release includes a bunch of small but useful new functions and some increased consistency with the rest of the tidyverse. [Read more ...](/blog/2022/12/stringr-1-5-0/) 2022/12/05 [![](/blog/2019/02/stringr-1-4-0/stringr-1-4-0-sq.jpg)](/blog/2019/02/stringr-1-4-0/) [stringr 1.4.0](/blog/2019/02/stringr-1-4-0/) [package](/categories/package) Mara Averick stringr 1.4.0 is now on CRAN! [Read more ...](/blog/2019/02/stringr-1-4-0/) 2019/02/21 [![](/blog/2018/02/stringr-1-3-0/stringr-1-3-0-sq.jpg)](/blog/2018/02/stringr-1-3-0/) [stringr 1.3.0](/blog/2018/02/stringr-1-3-0/) [package](/categories/package) Mara Averick stringr 1.3.0 is now on CRAN. [Read more ...](/blog/2018/02/stringr-1-3-0/) 2018/02/22 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # plots [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Plots [![](/blog/2022/11/model-calibration/thumbnail-sq.jpg)](/blog/2022/11/model-calibration/) [Model Calibration](/blog/2022/11/model-calibration/) [package](/categories/package) Edgar Ruiz Model Calibration is coming to tidymodels. This post covers the new plotting functions, and our plans for future enhancements. [Read more ...](/blog/2022/11/model-calibration/) 2022/11/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyclust is on CRAN [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) tidyclust is on CRAN ==================== ![](/blog/2022/12/tidyclust-0-1-0/thumbnail-wd.jpg) Photo by [Ankush Minda](https://unsplash.com/photos/4Xy08NbMBLM)   2022/12/06   [tidymodels](/tags/tidymodels/) , [tidyclust](/tags/tidyclust/)   Emil Hvitfeldt We’re very pleased to announce the release of [tidyclust](https://tidyclust.tidymodels.org/) 0.1.0. tidyclust is the tidymodels extension for working with clustering models. This package wouldn’t have been possible without the great work of [Kelly Bodwin](https://twitter.com/KellyBodwin) . You can install it from CRAN with: install.packages("tidyclust") This blog post will introduce tidyclust, how to use it with the rest of tidymodels, and how we can interact and evaluate the fitted clustering models. library(tidymodels) #> ── Attaching packages ────────────────────────────────────── tidymodels 1.0.0 ── #> ✔ broom 1.0.1 ✔ recipes 1.0.3 #> ✔ dials 1.1.0 ✔ rsample 1.1.0 #> ✔ dplyr 1.0.10 ✔ tibble 3.1.8 #> ✔ ggplot2 3.4.0 ✔ tidyr 1.2.1 #> ✔ infer 1.0.4 ✔ tune 1.0.1 #> ✔ modeldata 1.0.1 ✔ workflows 1.1.2 #> ✔ parsnip 1.0.3 ✔ workflowsets 1.0.0 #> ✔ purrr 0.3.5 ✔ yardstick 1.1.0 #> ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ── #> ✖ purrr::discard() masks scales::discard() #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag() #> ✖ recipes::step() masks stats::step() #> • Use suppressPackageStartupMessages() to eliminate package startup messages library(tidyclust) Specifying clustering models[](#specifying-clustering-models) -------------------------------------------------------------- The first thing we need to do is decide on the type of clustering model we want to fit. The pkgdown site provides a [list of all clustering specifications](https://tidyclust.tidymodels.org/reference/index.html#specifications) provided by tidyclust. We are slowly adding more types of models— [suggestions in issues](https://github.com/tidymodels/tidyclust/issues) are highly welcome! We will use a K-Means model for these examples using [`k_means()`](https://rdrr.io/pkg/tidyclust/man/k_means.html) to create a specification. As with other packages in the tidymodels, tidyclust tries to make use of informative names for functions and arguments; as such, the argument denoting the number of clusters is `num_clusters` rather than `k`. kmeans_spec <- k_means(num_clusters = 4) %>% set_engine("ClusterR") kmeans_spec #> K Means Cluster Specification (partition) #> #> Main Arguments: #> num_clusters = 4 #> #> Computational engine: ClusterR We can use the [`set_engine()`](https://parsnip.tidymodels.org/reference/set_engine.html) , [`set_mode()`](https://parsnip.tidymodels.org/reference/set_args.html) , and [`set_args()`](https://parsnip.tidymodels.org/reference/set_args.html) functions we are familiar with from parsnip. The specification itself isn’t worth much if we don’t apply it to some data. We will use the ames data set from the modeldata package. data("ames", package = "modeldata") This data set contains a number of categorical variables that unaltered can’t be used with a K-Means model. Some light preprocessing can be done using the recipes package. rec_spec <- recipe(~ ., data = ames) %>% step_dummy(all_nominal_predictors()) %>% step_zv(all_predictors()) %>% step_normalize(all_numeric_predictors()) %>% step_pca(all_numeric_predictors(), threshold = 0.8) This recipe normalizes all of the numeric variables before applying PCA to create a more minimal set of uncorrelated features. Notice how we didn’t specify an outcome as clustering models are unsupervised, meaning that we don’t have outcomes. These two specifications can be combined in a `workflow()`. kmeans_wf <- workflow(rec_spec, kmeans_spec) This workflow can then be fit to the `ames` data set. kmeans_fit <- fit(kmeans_wf, data = ames) kmeans_fit #> ══ Workflow [trained] ══════════════════════════════════════════════════════════ #> Preprocessor: Recipe #> Model: k_means() #> #> ── Preprocessor ──────────────────────────────────────────────────────────────── #> 4 Recipe Steps #> #> • step_dummy() #> • step_zv() #> • step_normalize() #> • step_pca() #> #> ── Model ─────────────────────────────────────────────────────────────────────── #> KMeans Cluster #> Call: ClusterR::KMeans_rcpp(data = data, clusters = clusters) #> Data cols: 121 #> Centroids: 4 #> BSS/SS: 0.1003306 #> SS: 646321.6 = 581475.8 (WSS) + 64845.81 (BSS) We have arbitrarily set the number of clusters to 4 above. If we wanted to figure out what values would be “optimal,” we would have to fit multiple models. We can do this with [`tune_cluster()`](https://rdrr.io/pkg/tidyclust/man/tune_cluster.html) ; to make use of this function, though, we first need to use [`tune()`](https://hardhat.tidymodels.org/reference/tune.html) to specify that `num_clusters` is the argument we want to try with multiple values. kmeans_spec <- kmeans_spec %>% set_args(num_clusters = tune()) kmeans_wf <- workflow(rec_spec, kmeans_spec) kmeans_wf #> ══ Workflow ════════════════════════════════════════════════════════════════════ #> Preprocessor: Recipe #> Model: k_means() #> #> ── Preprocessor ──────────────────────────────────────────────────────────────── #> 4 Recipe Steps #> #> • step_dummy() #> • step_zv() #> • step_normalize() #> • step_pca() #> #> ── Model ─────────────────────────────────────────────────────────────────────── #> K Means Cluster Specification (partition) #> #> Main Arguments: #> num_clusters = tune() #> #> Computational engine: ClusterR We can use [`tune_cluster()`](https://rdrr.io/pkg/tidyclust/man/tune_cluster.html) in the same way we use `tune_grid()`, using bootstraps to fit multiple models for each value of `num_clusters`. set.seed(1234) boots <- bootstraps(ames, times = 10) tune_res <- tune_cluster( kmeans_wf, resamples = boots ) The different [collect functions](https://tune.tidymodels.org/reference/collect_predictions.html) such as `collect_metrics()` works as they would do with tune output. collect_metrics(tune_res) #> # A tibble: 18 × 7 #> num_clusters .metric .estimator mean n std_err .config #> #> 1 6 sse_total standard 624435. 10 1675. Preprocessor1… #> 2 6 sse_within_total standard 557147. 10 2579. Preprocessor1… #> 3 1 sse_total standard 624435. 10 1675. Preprocessor1… #> 4 1 sse_within_total standard 624435. 10 1675. Preprocessor1… #> 5 3 sse_total standard 624435. 10 1675. Preprocessor1… #> 6 3 sse_within_total standard 588001. 10 5703. Preprocessor1… #> 7 5 sse_total standard 624435. 10 1675. Preprocessor1… #> 8 5 sse_within_total standard 568085. 10 3821. Preprocessor1… #> 9 9 sse_total standard 624435. 10 1675. Preprocessor1… #> 10 9 sse_within_total standard 535120. 10 2262. Preprocessor1… #> 11 2 sse_total standard 624435. 10 1675. Preprocessor1… #> 12 2 sse_within_total standard 599762. 10 4306. Preprocessor1… #> 13 8 sse_total standard 624435. 10 1675. Preprocessor1… #> 14 8 sse_within_total standard 541813. 10 2506. Preprocessor1… #> 15 4 sse_total standard 624435. 10 1675. Preprocessor1… #> 16 4 sse_within_total standard 583604. 10 5523. Preprocessor1… #> 17 7 sse_total standard 624435. 10 1675. Preprocessor1… #> 18 7 sse_within_total standard 548299. 10 2907. Preprocessor1… Extraction[](#extraction) -------------------------- Going back to the first model we fit, tidyclust provides three main tools for interfacing with a fitted cluster model: * extract cluster assignments * extract centroid locations * prediction with new data Each of these tasks has a function associated with them. First, we have [`extract_cluster_assignment()`](https://rdrr.io/pkg/tidyclust/man/extract_cluster_assignment.html) , which can be used on fitted tidyclust objects, alone or as a part of a workflow, and it returns the cluster assignment as a factor named `.cluster` in a tibble. extract_cluster_assignment(kmeans_fit) #> # A tibble: 2,930 × 1 #> .cluster #> #> 1 Cluster_1 #> 2 Cluster_1 #> 3 Cluster_1 #> 4 Cluster_1 #> 5 Cluster_2 #> 6 Cluster_2 #> 7 Cluster_2 #> 8 Cluster_2 #> 9 Cluster_2 #> 10 Cluster_2 #> # … with 2,920 more rows The location of the clusters can be found using [`extract_centroids()`](https://rdrr.io/pkg/tidyclust/man/extract_centroids.html) which again returns a tibble, with `.cluster` being a factor with the same levels as what we got from [`extract_cluster_assignment()`](https://rdrr.io/pkg/tidyclust/man/extract_cluster_assignment.html) . extract_centroids(kmeans_fit) #> # A tibble: 4 × 122 #> .cluster PC001 PC002 PC003 PC004 PC005 PC006 PC007 PC008 PC009 #> #> 1 Cluster_1 -5.76 0.713 11.9 2.80 4.09 3.44 1.26 -0.280 -0.486 #> 2 Cluster_2 3.98 -1.18 0.126 0.718 0.150 0.0554 -0.0460 -0.346 0.0599 #> 3 Cluster_3 -0.970 2.45 -0.604 -0.523 0.302 -0.298 -0.174 0.507 -0.153 #> 4 Cluster_4 -4.40 -2.30 -0.658 -0.671 -1.29 -0.00751 0.222 -0.250 0.223 #> # … with 112 more variables: PC010 , PC011 , PC012 , #> # PC013 , PC014 , PC015 , PC016 , PC017 , #> # PC018 , PC019 , PC020 , PC021 , PC022 , #> # PC023 , PC024 , PC025 , PC026 , PC027 , #> # PC028 , PC029 , PC030 , PC031 , PC032 , #> # PC033 , PC034 , PC035 , PC036 , PC037 , #> # PC038 , PC039 , PC040 , PC041 , PC042 , … Lastly, if the model has a notion that translates to “prediction,” then [`predict()`](https://rdrr.io/r/stats/predict.html) will give you those results as well. In the case of K-Means, this is being interpreted as “which centroid is this observation closest to.” predict(kmeans_fit, new_data = slice_sample(ames, n = 10)) #> # A tibble: 10 × 1 #> .pred_cluster #> #> 1 Cluster_4 #> 2 Cluster_2 #> 3 Cluster_4 #> 4 Cluster_3 #> 5 Cluster_1 #> 6 Cluster_4 #> 7 Cluster_2 #> 8 Cluster_2 #> 9 Cluster_1 #> 10 Cluster_4 Please check the [pkgdown site](https://tidyclust.tidymodels.org/) for more in-depth articles. We couldn’t be happier to have this package on CRAN and we encouraging you to check it out. Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all the contributors: [@aephidayatuloh](https://github.com/aephidayatuloh) , [@avishaitsur](https://github.com/avishaitsur) , [@bryanosborne](https://github.com/bryanosborne) , [@cgoo4](https://github.com/cgoo4) , [@coforfe](https://github.com/coforfe) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@JauntyJJS](https://github.com/JauntyJJS) , [@kbodwin](https://github.com/kbodwin) , [@malcolmbarrett](https://github.com/malcolmbarrett) , [@mattwarkentin](https://github.com/mattwarkentin) , [@ninohardt](https://github.com/ninohardt) , [@nipnipj](https://github.com/nipnipj) , and [@tomazweiss](https://github.com/tomazweiss) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # stringr 1.5.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) stringr 1.5.0 ============= ![](/blog/2022/12/stringr-1-5-0/thumbnail-wd.jpg) Photo by [Amie Bell](https://unsplash.com/photos/XGqS569rdgk)   2022/12/05   [tidyverse](/tags/tidyverse/) , [stringr](/tags/stringr/)   Hadley Wickham We’re chuffed to announce the release of [stringr](https://stringr.tidyverse.org) 1.5.0. stringr provides a cohesive set of functions designed to make working with strings as easy as possible. You can install it from CRAN with: install.packages("stringr") This blog post will give you an overview of the biggest changes (you can get a detailed list of all changes from the [release notes](https://stringr.tidyverse.org/news/index.html) ). Firstly, we need to update you on some (small) breaking changes we’ve made to make stringr more consistent with the rest of the tidyverse. Then, we’ll give a quick overview of improvements to documentation and stringr’s new license. Lastly, we’ll finish off by diving into a few of the many small, but useful, functions that we’ve accumulated in the three and half years since the last release. library(stringr) Breaking changes[](#breaking-changes) -------------------------------------- Lets start with the important stuff: the breaking changes. We’ve tried to keep these small and we don’t believe they’ll affect much code in the wild (they only affected ~20 of the ~1,600 packages that use stringr). But we’re believe they’re important to make as a consistent set of rules makes the tidyverse as a whole more predictable and easier to learn. ### Recycling rules[](#recycling-rules) stringr functions now consistently implement the tidyverse recycling rules[1](#fn:1) , which are stricter than the previous rules in two ways. Firstly, we no longer recycle shorter vectors that are an integer multiple of longer vectors: str_detect(letters, c("x", "y")) #> Error in `str_detect()`: #> ! Can't recycle `string` (size 26) to match `pattern` (size 2). Secondly, a 0-length vector no longer implies a 0-length output. Instead it’s recycled using the usual rules: str_detect(letters, character()) #> Error in `str_detect()`: #> ! Can't recycle `string` (size 26) to match `pattern` (size 0). str_detect("x", character()) #> logical(0) Neither of these situations occurs very commonly in data analysis, so this change primarily brings consistency with the rest of the tidyverse without affecting much existing code. Finally, stringr functions are generally a little stricter because we require the inputs to be vectors of some type. Again, this is unlikely to affect your data analysis code and will result in a clearer error if you accidentally pass in something weird: str_detect(mean, "x") #> Error in `str_detect()`: #> ! `string` must be a vector, not a function. ### Empty patterns[](#empty-patterns) In many stringr functions, `""` will match or split on every character. This is motivated by base R’s [`strsplit()`](https://rdrr.io/r/base/strsplit.html) : strsplit("abc", "")[[1]] #> [1] "a" "b" "c" str_split("abc", "")[[1]] #> [1] "a" "b" "c" When creating stringr (over 13 years ago!), I took this idea and ran with it, implementing similar support in every function where it might possibly work. But I missed an important problem with [`str_detect()`](https://stringr.tidyverse.org/reference/str_detect.html) . What should `str_detect(X, "")` return? You can argue two ways: * To be consistent with [`str_split()`](https://stringr.tidyverse.org/reference/str_split.html) , it should return `TRUE` whenever there are characters to match, i.e. `x != ""`. * It’s common to build up a set of possible matches by doing `str_flatten(matches, "|")`. What should this match if `matches` is empty? Ideally it would match nothing implying that `str_detect(x, "")` should be equivalent to `x == ""`. This inconsistency potentially leads to some subtle bugs, so use of `""` in [`str_detect()`](https://stringr.tidyverse.org/reference/str_detect.html) (and a few other related functions) is now an error: str_detect(letters, "") #> Error in `str_detect()`: #> ! `pattern` can't be the empty string (`""`). Documentation and licensing[](#documentation-and-licensing) ------------------------------------------------------------ Now that we’ve got the breaking changes out of the way we can focus on the new stuff 😃. Most importantly, there’s a new vignette that provides some advice if you’re transition from (or to) base R’s string functions: [`vignette("from-base", package = "stringr")`](https://stringr.tidyverse.org/articles/from-base.html) . It was written by [Sara Stoudt](https://sastoudt.github.io) during the 2019 Tidyverse developer day, and has finally made it to the released version! We’ve also spent a bunch of time reviewing the documentation, particularly the topic titles and descriptions. They’re now more informative and less duplicative, hopefully make it easier to find the function that you’re looking for. See the complete list of functions in the [reference index](https://stringr.tidyverse.org/reference/index.html) . Finally, stringr is now officially [re-licensed as MIT](https://www.tidyverse.org/blog/2021/12/relicensing-packages/) . New features[](#new-features) ------------------------------ The biggest improvement is to [`str_view()`](https://stringr.tidyverse.org/reference/str_view.html) which has gained a bunch of new features, including using the [cli](https://cli.r-lib.org/) package so it can work in more places. We also have a grab bag of new functions that fill in small functionality gaps. ### `str_view()`[](#str_view) [`str_view()`](https://stringr.tidyverse.org/reference/str_view.html) uses ANSI colouring rather than an HTML widget. This means it works in more places and requires fewer dependencies. [`str_view()`](https://stringr.tidyverse.org/reference/str_view.html) now: * Displays strings with special characters: x <- c("\\", "\"\nabcdef\n\"") x #> [1] "\\" "\"\nabcdef\n\"" str_view(x) #> [1] │ \ #> [2] │ " #> │ abcdef #> │ " * Highlights unusual whitespace characters: str_view("\t") #> [1] │ {\t} * By default, only shows matching strings: str_view(fruit, "(.)\\1") #> [1] │ ale #> [5] │ be peer #> [6] │ bilbey #> [7] │ blackbey #> [8] │ blackcuant #> [9] │ bld orange #> [10] │ bluebey #> [11] │ boysenbey #> [16] │ chey #> [17] │ chili peer #> [19] │ cloudbey #> [21] │ cranbey #> [23] │ cuant #> [28] │ eplant #> [29] │ elderbey #> [32] │ goji bey #> [33] │ gsebey #> [38] │ hucklebey #> [47] │ lych #> [50] │ mulbey #> ... and 9 more (This makes [`str_view_all()`](https://stringr.tidyverse.org/reference/str_view.html) redundant and hence deprecated.) ### Comparing strings[](#comparing-strings) There are three new functions related to comparing strings: * [`str_equal()`](https://stringr.tidyverse.org/reference/str_equal.html) compares two character vectors using Unicode rules, optionally ignoring case: str_equal("a", "A") #> [1] FALSE str_equal("a", "A", ignore_case = TRUE) #> [1] TRUE * [`str_rank()`](https://stringr.tidyverse.org/reference/str_order.html) completes the set of order/rank/sort functions: str_rank(c("a", "c", "b", "b")) #> [1] 1 4 2 2 # compare to: str_order(c("a", "c", "b", "b")) #> [1] 1 3 4 2 * [`str_unique()`](https://stringr.tidyverse.org/reference/str_unique.html) returns unique values, optionally ignoring case: str_unique(c("a", "a", "A")) #> [1] "a" "A" str_unique(c("a", "a", "A"), ignore_case = TRUE) #> [1] "a" ### Splitting[](#splitting) [`str_split()`](https://stringr.tidyverse.org/reference/str_split.html) gains two useful variants: * [`str_split_1()`](https://stringr.tidyverse.org/reference/str_split.html) is tailored for the special case of splitting up a single string. It returns a character vector, not a list, and errors if you try and give it multiple values: str_split_1("x-y-z", "-") #> [1] "x" "y" "z" str_split_1(c("x-y", "a-b-c"), "-") #> Error in `str_split_1()`: #> ! `string` must be a single string, not a character vector. It’s a shortcut for the common pattern of `unlist(str_split(x, " "))`. * [`str_split_i()`](https://stringr.tidyverse.org/reference/str_split.html) extracts a single piece from the split string: x <- c("a-b-c", "d-e", "f-g-h-i") str_split_i(x, "-", 2) #> [1] "b" "e" "g" str_split_i(x, "-", 4) #> [1] NA NA "i" str_split_i(x, "-", -1) #> [1] "c" "e" "i" ### Miscellaneous[](#miscellaneous) * [`str_escape()`](https://stringr.tidyverse.org/reference/str_escape.html) escapes regular expression metacharacters, providing an alternative to [`fixed()`](https://stringr.tidyverse.org/reference/modifiers.html) if you want to compose a pattern from user supplied strings: str_view("[hello]", str_escape("[]")) * [`str_extract()`](https://stringr.tidyverse.org/reference/str_extract.html) can now extract a capturing group instead of the complete match: x <- c("Chapter 1", "Section 2.3", "Chapter 3", "Section 4.1.1") str_extract(x, "([A-Za-z]+) ([0-9.]+)", group = 1) #> [1] "Chapter" "Section" "Chapter" "Section" str_extract(x, "([A-Za-z]+) ([0-9.]+)", group = 2) #> [1] "1" "2.3" "3" "4.1.1" * [`str_flatten()`](https://stringr.tidyverse.org/reference/str_flatten.html) gains a `last` argument which is used to power the new [`str_flatten_comma()`](https://stringr.tidyverse.org/reference/str_flatten.html) : str_flatten_comma(c("cats", "dogs", "mice")) #> [1] "cats, dogs, mice" str_flatten_comma(c("cats", "dogs", "mice"), last = " and ") #> [1] "cats, dogs and mice" str_flatten_comma(c("cats", "dogs", "mice"), last = ", and ") #> [1] "cats, dogs, and mice" # correctly handles the two element case with the Oxford comma str_flatten_comma(c("cats", "dogs"), last = ", and ") #> [1] "cats and dogs" * [`str_like()`](https://stringr.tidyverse.org/reference/str_like.html) works like [`str_detect()`](https://stringr.tidyverse.org/reference/str_detect.html) but uses SQL’s LIKE syntax: fruit <- c("apple", "banana", "pear", "pineapple") fruit[str_like(fruit, "%apple")] #> [1] "apple" "pineapple" fruit[str_like(fruit, "p__r")] #> [1] "pear" Acknowledgements[](#acknowledgements) -------------------------------------- A big thanks to all 114 folks who contributed to this release through pull requests and issues! [@aaronrudkin](https://github.com/aaronrudkin) , [@adisarid](https://github.com/adisarid) , [@AleSR13](https://github.com/AleSR13) , [@anfederico](https://github.com/anfederico) , [@AR1337](https://github.com/AR1337) , [@arisp99](https://github.com/arisp99) , [@avila](https://github.com/avila) , [@balthasars](https://github.com/balthasars) , [@batpigandme](https://github.com/batpigandme) , [@bbarros50](https://github.com/bbarros50) , [@bbo2adwuff](https://github.com/bbo2adwuff) , [@bensenmansen](https://github.com/bensenmansen) , [@bfgray3](https://github.com/bfgray3) , [@Bisaloo](https://github.com/Bisaloo) , [@bonmac](https://github.com/bonmac) , [@botan](https://github.com/botan) , [@bshor](https://github.com/bshor) , [@carlganz](https://github.com/carlganz) , [@chintanp](https://github.com/chintanp) , [@chrimaho](https://github.com/chrimaho) , [@chris2b5](https://github.com/chris2b5) , [@clemenshug](https://github.com/clemenshug) , [@courtiol](https://github.com/courtiol) , [@dachosen1](https://github.com/dachosen1) , [@dan-reznik](https://github.com/dan-reznik) , [@datawookie](https://github.com/datawookie) , [@david-romano](https://github.com/david-romano) , [@DavisVaughan](https://github.com/DavisVaughan) , [@dbarrows](https://github.com/dbarrows) , [@deann88](https://github.com/deann88) , [@denrou](https://github.com/denrou) , [@deschen1](https://github.com/deschen1) , [@dsg38](https://github.com/dsg38) , [@dtburk](https://github.com/dtburk) , [@elbersb](https://github.com/elbersb) , [@geotheory](https://github.com/geotheory) , [@ghost](https://github.com/ghost) , [@GrimTrigger88](https://github.com/GrimTrigger88) , [@hadley](https://github.com/hadley) , [@iago-pssjd](https://github.com/iago-pssjd) , [@IndigoJay](https://github.com/IndigoJay) , [@jashapiro](https://github.com/jashapiro) , [@JBGruber](https://github.com/JBGruber) , [@jennybc](https://github.com/jennybc) , [@jimhester](https://github.com/jimhester) , [@jjesusfilho](https://github.com/jjesusfilho) , [@jmbarbone](https://github.com/jmbarbone) , [@joethorley](https://github.com/joethorley) , [@jonas-hag](https://github.com/jonas-hag) , [@jonthegeek](https://github.com/jonthegeek) , [@joshyam-k](https://github.com/joshyam-k) , [@jpeacock29](https://github.com/jpeacock29) , [@jzadra](https://github.com/jzadra) , [@KasperThystrup](https://github.com/KasperThystrup) , [@kendonB](https://github.com/kendonB) , [@kieran-mace](https://github.com/kieran-mace) , [@kiernann](https://github.com/kiernann) , [@Kodiologist](https://github.com/Kodiologist) , [@leej3](https://github.com/leej3) , [@leowill01](https://github.com/leowill01) , [@LimaRAF](https://github.com/LimaRAF) , [@lmwang9527](https://github.com/lmwang9527) , [@Ludsfer](https://github.com/Ludsfer) , [@lz01](https://github.com/lz01) , [@Marcade80](https://github.com/Marcade80) , [@Mashin6](https://github.com/Mashin6) , [@MattCowgill](https://github.com/MattCowgill) , [@maxheld83](https://github.com/maxheld83) , [@mgirlich](https://github.com/mgirlich) , [@MichaelChirico](https://github.com/MichaelChirico) , [@michaelweylandt](https://github.com/michaelweylandt) , [@mikeaalv](https://github.com/mikeaalv) , [@misea](https://github.com/misea) , [@mitchelloharawild](https://github.com/mitchelloharawild) , [@mkvasnicka](https://github.com/mkvasnicka) , [@mrcaseb](https://github.com/mrcaseb) , [@mtnbikerjoshua](https://github.com/mtnbikerjoshua) , [@mwip](https://github.com/mwip) , [@nachovoss](https://github.com/nachovoss) , [@neonira](https://github.com/neonira) , [@Nischal-Karki-ATW](https://github.com/Nischal-Karki-ATW) , [@oliverbeagley](https://github.com/oliverbeagley) , [@orgadish](https://github.com/orgadish) , [@pachadotdev](https://github.com/pachadotdev) , [@PathosEthosLogos](https://github.com/PathosEthosLogos) , [@pdelboca](https://github.com/pdelboca) , [@petermeissner](https://github.com/petermeissner) , [@phargarten2](https://github.com/phargarten2) , [@programLyrique](https://github.com/programLyrique) , [@psads-git](https://github.com/psads-git) , [@psychelzh](https://github.com/psychelzh) , [@PursuitOfDataScience](https://github.com/PursuitOfDataScience) , [@richardjtelford](https://github.com/richardjtelford) , [@richelbilderbeek](https://github.com/richelbilderbeek) , [@rjpat](https://github.com/rjpat) , [@romatik](https://github.com/romatik) , [@rressler](https://github.com/rressler) , [@rwbaer](https://github.com/rwbaer) , [@salim-b](https://github.com/salim-b) , [@sammo3182](https://github.com/sammo3182) , [@sastoudt](https://github.com/sastoudt) , [@SchmidtPaul](https://github.com/SchmidtPaul) , [@seasmith](https://github.com/seasmith) , [@selesnow](https://github.com/selesnow) , [@slee981](https://github.com/slee981) , [@Tal1987](https://github.com/Tal1987) , [@tanzatanza](https://github.com/tanzatanza) , [@THChan11](https://github.com/THChan11) , [@travis-leith](https://github.com/travis-leith) , [@vladtarko](https://github.com/vladtarko) , [@wdenton](https://github.com/wdenton) , [@wurli](https://github.com/wurli) , [@Yingjie4Science](https://github.com/Yingjie4Science) , and [@zeehio](https://github.com/zeehio) . * * * 1. You might wonder why we developed our own set of recycling rules for the tidyverse instead of using the base R rules. That’s because, unfortunately, there isn’t a consistent set of rules used by base R, but a [suite of variations](https://vctrs.r-lib.org/articles/type-size.html#appendix-recycling-in-base-r) . [↩︎](#fnref:1) Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # agua [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Agua [![](/blog/2022/10/tidymodels-2022-q3/thumbnail-sq.jpg)](/blog/2022/10/tidymodels-2022-q3/) [Q3 2022 tidymodels digest](/blog/2022/10/tidymodels-2022-q3/) [roundup](/categories/roundup) Max Kuhn Our post-RStudio conference productivity has been high! This post talks about tidymodels updates from the last few months. [Read more ...](/blog/2022/10/tidymodels-2022-q3/) 2022/10/19 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # purrr 1.0.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) purrr 1.0.0 =========== ![](/blog/2022/12/purrr-1-0-0/thumbnail-wd.jpg) Photo by [Jari Hytönen](https://unsplash.com/photos/YCPkW_r_6uA)   2022/12/20   [purrr](/tags/purrr/)   Hadley Wickham We’re happy to announce the release of [purrr](http://purrr.tidyverse.org/) 1.0.0! purrr enhances R’s functional programming toolkit by providing a complete and consistent set of tools for working with functions and vectors. In the words of ChatGPT: > With purrr, you can easily “kitten” your functions together to perform complex operations, “paws” for a moment to debug and troubleshoot your code, while “feline” good about the elegant and readable code that you write. Whether you’re a “cat”-egorical beginner or a seasoned functional programming “purr”-fessional, purrr has something to offer. So why not “pounce” on the opportunity to try it out and see how it can “meow”-velously improve your R coding experience? You can install it from CRAN with: install.packages("purrr") purrr is 7 years old and it’s finally made it to 1.0.0! This is a big release, adding some long-needed functionality (like progress bars!) as well as really refining the core purpose of purrr. In this post, we’ll start with an overview of the breaking changes, then briefly review some documentation changes. Then we’ll get to the good stuff: improvements to the `map` family, new [`keep_at()`](https://purrr.tidyverse.org/reference/keep_at.html) and [`discard_at()`](https://purrr.tidyverse.org/reference/keep_at.html) functions, and improvements to flattening and simplification. You can see a full list of changes in the [release notes](https://github.com/tidyverse/purrr/releases/tag/v1.0.0) . library(purrr) Breaking changes[](#breaking-changes) -------------------------------------- We’ve used the 1.0.0 release as an opportunity to really refine the core purpose of purrr: facilitating functional programming in R. We’ve been more aggressive with deprecations and breaking changes than usual, because a 1.0.0 release signals that purrr is now [stable](https://lifecycle.r-lib.org/articles/stages.html#stable) , making it our last opportunity for major changes. These changes will break some existing code, but we’ve done our best to make it affect as little code as possible. Out of the ~1400 CRAN packages that user purrr, only ~40 were negatively affected, and I [made pull requests](https://github.com/tidyverse/purrr/issues/969) to fix them all. Making these fixes helped give me confidence that, though we’re deprecating quite a few functions and changing a few special cases, it shouldn’t affect too much code in the wild. There are four important changes that you should be aware of: * [`pluck()`](https://purrr.tidyverse.org/reference/pluck.html) behaves differently when extracting 0-length vectors. * The [`map()`](https://purrr.tidyverse.org/reference/map.html) family uses the tidyverse rules for coercion and recycling. * All functions that modify lists handle `NULL` consistently. * We’ve deprecated functions that aren’t related to the core purpose of purrr. ### `pluck()` and zero-length vectors[](#pluck-and-zero-length-vectors) Previously, [`pluck()`](https://purrr.tidyverse.org/reference/pluck.html) replaced 0-length vectors with the value of `default`. Now `default` is only used for `NULL`s and absent elements: x <- list(y = list(a = character(), b = NULL)) x |> pluck("y", "a", .default = NA) #> character(0) x |> pluck("y", "b", .default = NA) #> [1] NA x |> pluck("y", "c", .default = NA) #> [1] NA This also influences the map family because using an integer vector, character vector, or list instead of a function automatically calls [`pluck()`](https://purrr.tidyverse.org/reference/pluck.html) : x <- list(list(1), list(), list(NULL), list(character())) x |> map(1, .default = 0) |> str() #> List of 4 #> $ : num 1 #> $ : num 0 #> $ : num 0 #> $ : chr(0) We made this change because it makes purrr more consistent with the rest of the tidyverse and it looks like it was a bug in the original implementation of the function. ### Tidyverse consistency[](#tidyverse-consistency) We’ve tweaked the map family of functions to be more consistent with general tidyverse coercion and recycling rules, as implemented by the [vctrs](https://vctrs.r-lib.org) package. [`map_lgl()`](https://purrr.tidyverse.org/reference/map.html) , [`map_int()`](https://purrr.tidyverse.org/reference/map.html) , [`map_int()`](https://purrr.tidyverse.org/reference/map.html) , and [`map_dbl()`](https://purrr.tidyverse.org/reference/map.html) now follow the same [coercion rules](https://vctrs.r-lib.org/articles/type-size.html#coercing-to-common-type) as vctrs. In particular: * `map_chr(TRUE, identity)`, `map_chr(0L, identity)`, and `map_chr(1.5, identity)` have been deprecated because we believe that converting a logical/integer/double to a character vector is potentially dangerous and should require an explicit coercion. # previously you could write map_chr(1:4, \(x) x + 1) #> Warning: Automatic coercion from double to character was deprecated in purrr 1.0.0. #> ℹ Please use an explicit call to `as.character()` within `map_chr()` instead. #> [1] "2.000000" "3.000000" "4.000000" "5.000000" # now you need something like this: map_chr(1:4, \(x) as.character(x + 1)) #> [1] "2" "3" "4" "5" * [`map_int()`](https://purrr.tidyverse.org/reference/map.html) requires that the numeric results be close to integers, rather than silently truncating to integers. Compare these two examples: map_int(1:3, \(x) x / 2) #> Error in `map_int()`: #> ℹ In index: 1. #> Caused by error: #> ! Can't coerce from a double vector to an integer vector. map_int(1:3, \(x) x * 2) #> [1] 2 4 6 [`map2()`](https://purrr.tidyverse.org/reference/map2.html) , [`modify2()`](https://purrr.tidyverse.org/reference/modify.html) , and [`pmap()`](https://purrr.tidyverse.org/reference/pmap.html) use tidyverse recycling rules, which mean that vectors of length 1 are recycled to any size but all other vectors must have the same length. This has two major changes: * Previously, the presence of a zero-length input generated a zero-length output. Now it’s recycled using the same rules: map2(1:2, character(), paste) #> Error in `map2()`: #> ! Can't recycle `.x` (size 2) to match `.y` (size 0). # Works because length-1 vector gets recycled to length-0 map2(1, character(), paste) #> list() * And now must explicitly recycle vectors that aren’t length 1: map2_int(1:4, c(10, 20), `+`) #> Error in `map2_int()`: #> ! Can't recycle `.x` (size 4) to match `.y` (size 2). map2_int(1:4, rep(c(10, 20), 2), `+`) #> [1] 11 22 13 24 ### Assigning `NULL`[](#assigning-null) purrr has a number of functions that modify a list: `pluck<-()`, [`assign_in()`](https://purrr.tidyverse.org/reference/modify_in.html) , [`modify()`](https://purrr.tidyverse.org/reference/modify.html) , [`modify2()`](https://purrr.tidyverse.org/reference/modify.html) , [`modify_if()`](https://purrr.tidyverse.org/reference/modify.html) , [`modify_at()`](https://purrr.tidyverse.org/reference/modify.html) , and [`list_modify()`](https://purrr.tidyverse.org/reference/list_assign.html) . Previously, these functions had inconsistent behaviour when you attempted to modify an element with `NULL`: some functions would delete that element, and some would set it to `NULL`. That inconsistency arose because base R handles `NULL` in different ways depending on whether or not use you `$`/`[[` or `[`:\ \ x1 <- x2 <- x3 <- list(a = 1, b = 2)\ \ x1$a <- NULL\ str(x1)\ #> List of 1\ #> $ b: num 2\ \ x2["a"] <- list(NULL)\ str(x2)\ #> List of 2\ #> $ a: NULL\ #> $ b: num 2\ \ \ Now functions that edit a list will create an element containing `NULL`:\ \ x3 |> \ list_modify(a = NULL) |> \ str()\ #> List of 2\ #> $ a: NULL\ #> $ b: num 2\ \ x3 |> \ modify_at("b", \(x) NULL) |> \ str()\ #> List of 2\ #> $ a: num 1\ #> $ b: NULL\ \ \ If you want to delete the element, you can use the special [`zap()`](https://rlang.r-lib.org/reference/zap.html)\ sentinel:\ \ x3 |> \ list_modify(a = zap()) |> \ str()\ #> List of 1\ #> $ b: num 2\ \ \ [`zap()`](https://rlang.r-lib.org/reference/zap.html)\ does not work in `modify*()` because those functions are designed to always return the same top-level structure as the input.\ \ ### Core purpose refinements[](#core-purpose-refinements)\ \ We have **deprecated** a number of functions to keep purrr focused on its core purpose: facilitating functional programming in R. Deprecation means that the functions will continue to work, but you’ll be warned once every 8 hours if you use them. In several years time, we’ll release an update which causes the warnings to occur on every time you use them, and a few years after that they’ll be transformed to throwing errors.\ \ * [`cross()`](https://purrr.tidyverse.org/reference/cross.html)\ and all its variants have been deprecated because they’re slow and buggy, and a better approach already exists in [`tidyr::expand_grid()`](https://tidyr.tidyverse.org/reference/expand_grid.html)\ .\ \ * [`update_list()`](https://purrr.tidyverse.org/reference/update_list.html)\ , [`rerun()`](https://purrr.tidyverse.org/reference/rerun.html)\ , and the use of tidyselect with [`map_at()`](https://purrr.tidyverse.org/reference/map_if.html)\ and friends have been deprecated because we no longer believe that non-standard evaluation is a good fit for purrr.\ \ * The `lift_*` family of functions has been superseded because they promote a style of function manipulation that is not commonly used in R.\ \ * [`prepend()`](https://purrr.tidyverse.org/reference/prepend.html)\ , [`rdunif()`](https://purrr.tidyverse.org/reference/rdunif.html)\ , [`rbernoulli()`](https://purrr.tidyverse.org/reference/rbernoulli.html)\ , [`when()`](https://purrr.tidyverse.org/reference/when.html)\ , and [`list_along()`](https://purrr.tidyverse.org/reference/along.html)\ have been deprecated because they’re not directly related to functional programming.\ \ * `splice()` has been deprecated because we no longer believe that automatic splicing makes for good UI and there are other ways to achieve the same result.\ \ \ Consult the documentation for the alternatives that we now recommend.\ \ Deprecating these functions makes purrr easier to maintain because it reduces the surface area for bugs and issues, and it makes purrr easier to learn because there’s a clearer common thread that ties together all functions.\ \ Documentation[](#documentation)\ \ --------------------------------\ \ As you’ve seen in the code above, we are moving from magrittr’s pipe (`%>%`) to the base pipe (`|>`) and from formula syntax (`~ .x + 1`) to R’s new anonymous function short hand (`\(x) x + 1`). We believe that it’s better to use these new base tools because they work everywhere: the base pipe doesn’t require that you load magrittr and the new function shorthand works everywhere, not just in purrr functions. Additionally, being able to specify the argument name for the anonymous function can often lead to clearer code.\ \ # Previously we wrote\ 1:10 %>%\ map(~ rnorm(10, .x)) %>%\ map_dbl(mean)\ #> [1] 0.5586355 1.8213041 2.8764412 4.1521664 5.1160393 6.1271905\ #> [7] 6.9109806 8.2808301 9.2373940 10.6269104\ \ # Now we recommend\ 1:10 |>\ map(\(mu) rnorm(10, mu)) |>\ map_dbl(mean) \ #> [1] 0.4638639 2.0966712 3.4441928 3.7806185 5.3373228 6.1854820\ #> [7] 6.5873300 8.3116138 9.4824697 10.4590034\ \ \ We also recommend using an anonymous function instead of passing additional arguments to map. This avoids a certain class of moderately esoteric argument matching woes and, we believe, is generally easier to read.\ \ mu <- c(1, 10, 100)\ \ # Previously we wrote\ mu |> map_dbl(rnorm, n = 1)\ #> [1] 0.5706199 11.3604613 99.9291426\ \ # Now we recommend\ mu |> map_dbl(\(mu) rnorm(1, mean = mu))\ #> [1] 0.7278463 7.5533200 100.0654866\ \ \ Due to the [tidyverse R dependency policy](https://www.tidyverse.org/blog/2019/04/r-version-support/)\ , purrr works in R 3.5, 3.6, 4.0, 4.1, and 4.2, but the base pipe and anonymous function syntax are only available in R 4.0 and later. So the examples are automatically disabled on R 3.5 and 3.6 to allow purrr to continue to pass `R CMD check`.\ \ Mapping[](#mapping)\ \ --------------------\ \ With that out of the way, we can now talk about the exciting new features in purrr 1.0.0. We’ll start with the map family of functions which have three big new features:\ \ * Progress bars.\ * Better errors.\ * A new family member: [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ .\ \ These are described in the following sections.\ \ ### Progress bars[](#progress-bars)\ \ The map family can now produce a progress bar. This is very useful for long running jobs:\ \ ![](figs//progress.svg)\ \ (For interactive use, the progress bar uses some simple heuristics so that it doesn’t show up for very simple jobs.)\ \ In most cases, we expect that `.progress = TRUE` is enough, but if you’re wrapping [`map()`](https://purrr.tidyverse.org/reference/map.html)\ in another function, you might want to set `.progress` to a string that identifies the progress bar:\ \ ![](figs//named-progress.svg)\ \ ### Better errors[](#better-errors)\ \ If there’s an error in the function you’re mapping, [`map()`](https://purrr.tidyverse.org/reference/map.html)\ and friends now tell you which element caused the problem:\ \ x <- sample(1:500)\ x |> map(\(x) if (x == 1) stop("Error!") else 10)\ #> Error in `map()`:\ #> ℹ In index: 51.\ #> Caused by error in `.f()`:\ #> ! Error!\ \ \ We hope that this makes your debugging life just a little bit easier! (Don’t forget about [`safely()`](https://purrr.tidyverse.org/reference/safely.html)\ and [`possibly()`](https://purrr.tidyverse.org/reference/possibly.html)\ if you expect failures and want to either ignore or capture them.)\ \ We have also generally reviewed the error messages throughout purrr in order to make them more actionable. If you hit a confusing error message, please let us know!\ \ ### New `map_vec()`[](#new-map_vec)\ \ We’ve added [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ (along with [`map2_vec()`](https://purrr.tidyverse.org/reference/map2.html)\ , and [`pmap_vec()`](https://purrr.tidyverse.org/reference/pmap.html)\ ) to handle more types of vectors. [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ extends [`map_lgl()`](https://purrr.tidyverse.org/reference/map.html)\ , [`map_int()`](https://purrr.tidyverse.org/reference/map.html)\ , [`map_dbl()`](https://purrr.tidyverse.org/reference/map.html)\ , and [`map_chr()`](https://purrr.tidyverse.org/reference/map.html)\ to arbitrary types of vectors, like dates, factors, and date-times:\ \ 1:3 |> map_vec(\(i) factor(letters[i]))\ #> [1] a b c\ #> Levels: a b c\ 1:3 |> map_vec(\(i) factor(letters[i], levels = letters[4:1]))\ #> [1] a b c\ #> Levels: d c b a\ \ 1:3 |> map_vec(\(i) as.Date(ISOdate(i + 2022, 10, 5)))\ #> [1] "2023-10-05" "2024-10-05" "2025-10-05"\ 1:3 |> map_vec(\(i) ISOdate(i + 2022, 10, 5))\ #> [1] "2023-10-05 12:00:00 GMT" "2024-10-05 12:00:00 GMT"\ #> [3] "2025-10-05 12:00:00 GMT"\ \ \ [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ exists somewhat in the middle of base R’s [`sapply()`](https://rdrr.io/r/base/lapply.html)\ and [`vapply()`](https://rdrr.io/r/base/lapply.html)\ . Unlike [`sapply()`](https://rdrr.io/r/base/lapply.html)\ it will always return a simpler vector, erroring if there’s no common type:\ \ list("a", 1) |> map_vec(identity)\ #> Error in `map_vec()`:\ #> ! Can't combine `[[1]]` and `[[2]]` .\ \ \ If you want to require a certain type of output, supply `.ptype`, making [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ behave more like [`vapply()`](https://rdrr.io/r/base/lapply.html)\ . `ptype` is short for prototype, and should be a vector that exemplifies the type of output you expect.\ \ x <- list("a", "b") \ x |> map_vec(identity, .ptype = character())\ #> [1] "a" "b"\ \ # will error if the result can't be automatically coerced\ # to the specified ptype\ x |> map_vec(identity, .ptype = integer())\ #> Error in `map_vec()`:\ #> ! Can't convert `[[1]]` to .\ \ \ We don’t expect you to know or memorise the [rules that vctrs uses for coercion](https://vctrs.r-lib.org/reference/faq-compatibility-types.html)\ ; our hope is that they’ll become second nature as we steadily ensure that every tidyverse function follows the same rules.\ \ `keep_at()` and `discard_at()`[](#keep_at-and-discard_at)\ \ ----------------------------------------------------------\ \ purrr has gained a new pair of functions, [`keep_at()`](https://purrr.tidyverse.org/reference/keep_at.html)\ and [`discard_at()`](https://purrr.tidyverse.org/reference/keep_at.html)\ , that work like [`keep()`](https://purrr.tidyverse.org/reference/keep.html)\ and [`discard()`](https://purrr.tidyverse.org/reference/keep.html)\ but operate on names rather than values:\ \ x <- list(a = 1, b = 2, c = 3, D = 4, E = 5)\ \ x |> \ keep_at(c("a", "b", "c")) |> \ str()\ #> List of 3\ #> $ a: num 1\ #> $ b: num 2\ #> $ c: num 3\ \ x |> \ discard_at(c("a", "b", "c")) |> \ str()\ #> List of 2\ #> $ D: num 4\ #> $ E: num 5\ \ \ Alternatively, you can supply a function that is called with the names of the elements and should return a logical vector describing which elements to keep/discard:\ \ is_lower_case <- function(x) x == tolower(x)\ \ x |> keep_at(is_lower_case)\ #> $a\ #> [1] 1\ #> \ #> $b\ #> [1] 2\ #> \ #> $c\ #> [1] 3\ \ \ You can now also pass such a function to all other `_at()` functions:\ \ x |> \ modify_at(is_lower_case, \(x) x * 100) |> \ str()\ #> List of 5\ #> $ a: num 100\ #> $ b: num 200\ #> $ c: num 300\ #> $ D: num 4\ #> $ E: num 5\ \ \ Flattening and simplification[](#flattening-and-simplification)\ \ ----------------------------------------------------------------\ \ Last, but not least, we’ve reworked the family of functions that flatten and simplify lists. These caused us a lot of confusion internally because folks (and different packages) used the same words to mean different things. Now there are three main functions that share a common prefix that makes it clear that they all operate on lists:\ \ * [`list_flatten()`](https://purrr.tidyverse.org/reference/list_flatten.html)\ removes a single level of hierarchy from a list; the output is always a list.\ * [`list_simplify()`](https://purrr.tidyverse.org/reference/list_simplify.html)\ reduces a list to a homogeneous vector; the output is always the same length as the input.\ * [`list_c()`](https://purrr.tidyverse.org/reference/list_c.html)\ , [`list_cbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ , and [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ concatenate the elements of a list to produce a vector or data frame. There are no constraints on the output.\ \ These functions have lead us to **supersede** a number of functions. This means that they are not going away but we no longer recommend them, and they will receive only critical bug fixes.\ \ * `flatten()` has been superseded by [`list_flatten()`](https://purrr.tidyverse.org/reference/list_flatten.html)\ .\ * `flatten_lgl()`, `flatten_int()`, `flatten_dbl()`, and `flatten_chr()` have been superseded by [`list_c()`](https://purrr.tidyverse.org/reference/list_c.html)\ .\ * [`flatten_dfr()`](https://purrr.tidyverse.org/reference/flatten.html)\ and [`flatten_dfc()`](https://purrr.tidyverse.org/reference/flatten.html)\ have been superseded by [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ and [`list_cbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ respectively. [`flatten_dfr()`](https://purrr.tidyverse.org/reference/flatten.html)\ had some particularly puzzling edge cases when the inputs would be flattened into columns.\ * [`map_dfc()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ and [`map_dfr()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ (and their `map2` and `pmap` variants) have been superseded in favour of using the appropriate map function along with [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ or [`list_cbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ .\ * [`simplify()`](https://purrr.tidyverse.org/reference/as_vector.html)\ , [`simplify_all()`](https://purrr.tidyverse.org/reference/as_vector.html)\ , and [`as_vector()`](https://purrr.tidyverse.org/reference/as_vector.html)\ have been superseded in favour of [`list_simplify()`](https://purrr.tidyverse.org/reference/list_simplify.html)\ .\ \ ### Flattening[](#flattening)\ \ [`list_flatten()`](https://purrr.tidyverse.org/reference/list_flatten.html)\ removes one layer of hierarchy from a list. In other words, if any of the children of the list are themselves lists, the contents of those lists are inlined into the parent:\ \ x <- list(1, list(2, list(3, 4), 5))\ x |> str()\ #> List of 2\ #> $ : num 1\ #> $ :List of 3\ #> ..$ : num 2\ #> ..$ :List of 2\ #> .. ..$ : num 3\ #> .. ..$ : num 4\ #> ..$ : num 5\ x |> list_flatten() |> str()\ #> List of 4\ #> $ : num 1\ #> $ : num 2\ #> $ :List of 2\ #> ..$ : num 3\ #> ..$ : num 4\ #> $ : num 5\ x |> list_flatten() |> list_flatten() |> str()\ #> List of 5\ #> $ : num 1\ #> $ : num 2\ #> $ : num 3\ #> $ : num 4\ #> $ : num 5\ \ \ [`list_flatten()`](https://purrr.tidyverse.org/reference/list_flatten.html)\ always returns a list; once a list is as flat as it can get (i.e. none of its children contain lists), it leaves the input unchanged.\ \ x |> list_flatten() |> list_flatten() |> list_flatten() |> str()\ #> List of 5\ #> $ : num 1\ #> $ : num 2\ #> $ : num 3\ #> $ : num 4\ #> $ : num 5\ \ \ ### Simplification[](#simplification)\ \ [`list_simplify()`](https://purrr.tidyverse.org/reference/list_simplify.html)\ maintains the length of the input, but produces a simpler type:\ \ list(1, 2, 3) |> list_simplify()\ #> [1] 1 2 3\ list("a", "b", "c") |> list_simplify()\ #> [1] "a" "b" "c"\ \ \ Because the length must stay the same, it will only succeed if every element has length 1:\ \ list_simplify(list(1, 2, 3:4))\ #> Error in `list_simplify()`:\ #> ! `x[[3]]` must have size 1, not size 2.\ list_simplify(list(1, 2, integer()))\ #> Error in `list_simplify()`:\ #> ! `x[[3]]` must have size 1, not size 0.\ \ \ Because the result must be a simpler vector, all the components must be compatible:\ \ list_simplify(list(1, 2, "a"))\ #> Error in `list_simplify()`:\ #> ! Can't combine `[[1]]` and `[[3]]` .\ \ \ If you need to simplify if it’s possible, but otherwise leave the input unchanged, use `strict = FALSE`:\ \ list_simplify(list(1, 2, "a"), strict = FALSE)\ #> [[1]]\ #> [1] 1\ #> \ #> [[2]]\ #> [1] 2\ #> \ #> [[3]]\ #> [1] "a"\ \ \ If you want to be specific about the type you want, [`list_simplify()`](https://purrr.tidyverse.org/reference/list_simplify.html)\ can take the same prototype argument as [`map_vec()`](https://purrr.tidyverse.org/reference/map.html)\ :\ \ list(1, 2, 3) |> list_simplify(ptype = integer())\ #> [1] 1 2 3\ \ list(1, 2, 3) |> list_simplify(ptype = factor())\ #> Error in `list_simplify()`:\ #> ! Can't convert `[[1]]` to >.\ \ \ ### Concatenation[](#concatenation)\ \ [`list_c()`](https://purrr.tidyverse.org/reference/list_c.html)\ , [`list_cbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ , and [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ concatenate all elements together in a similar way to using `do.call(c)` or `do.call(rbind)`[1](#fn:1)\ . Unlike [`list_simplify()`](https://purrr.tidyverse.org/reference/list_simplify.html)\ , this allows the elements to be different lengths:\ \ list(1, 2, 3) |> list_c()\ #> [1] 1 2 3\ list(1, 2, 3:4, integer()) |> list_c()\ #> [1] 1 2 3 4\ \ \ The downside of this flexibility is that these functions break the connection between the input and the output. This reveals that [`map_dfr()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ and [`map_dfc()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ don’t really belong to the map family because they don’t maintain a 1-to-1 mapping between input and output: there’s reliable no way to associate a row in the output with an element in an input.\ \ For this reason, [`map_dfr()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ and [`map_dfc()`](https://purrr.tidyverse.org/reference/map_dfr.html)\ (and the `map2` and `pmap`) variants are superseded and we recommend switching to an explicit call to [`list_rbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ or [`list_cbind()`](https://purrr.tidyverse.org/reference/list_c.html)\ instead:\ \ paths |> map_dfr(read_csv, .id = "path")\ # now\ paths |> \ map(read_csv) |> \ list_rbind(names_to = "path")\ \ This new behaviour also affects to [`accumulate()`](https://purrr.tidyverse.org/reference/accumulate.html)\ and [`accumulate2()`](https://purrr.tidyverse.org/reference/accumulate.html)\ , which previously had an idiosyncratic approach to simplification.\ \ ### `list_assign()`[](#list_assign)\ \ There’s one other new function that isn’t directly related to flattening and friends, but shares the `list_` prefix: [`list_assign()`](https://purrr.tidyverse.org/reference/list_assign.html)\ . [`list_assign()`](https://purrr.tidyverse.org/reference/list_assign.html)\ is similar to [`list_modify()`](https://purrr.tidyverse.org/reference/list_assign.html)\ but it doesn’t work recursively. This is a mildly confusing feature of [`list_modify()`](https://purrr.tidyverse.org/reference/list_assign.html)\ that it’s easy to miss in the documentation.\ \ list(x = 1, y = list(a = 1)) |> \ list_modify(y = list(b = 1)) |> \ str()\ #> List of 2\ #> $ x: num 1\ #> $ y:List of 2\ #> ..$ a: num 1\ #> ..$ b: num 1\ \ \ [`list_assign()`](https://purrr.tidyverse.org/reference/list_assign.html)\ doesn’t recurse into sublists making it a bit easier to reason about:\ \ list(x = 1, y = list(a = 1)) |> \ list_assign(y = list(b = 2)) |> \ str()\ #> List of 2\ #> $ x: num 1\ #> $ y:List of 1\ #> ..$ b: num 2\ \ \ Acknowledgements[](#acknowledgements)\ \ --------------------------------------\ \ A massive thanks to all 162 contributors who have helped make purrr 1.0.0 happen! [@adamroyjones](https://github.com/adamroyjones)\ , [@afoltzm](https://github.com/afoltzm)\ , [@agilebean](https://github.com/agilebean)\ , [@ahjames11](https://github.com/ahjames11)\ , [@AHoerner](https://github.com/AHoerner)\ , [@alberto-dellera](https://github.com/alberto-dellera)\ , [@alex-gable](https://github.com/alex-gable)\ , [@AliciaSchep](https://github.com/AliciaSchep)\ , [@ArtemSokolov](https://github.com/ArtemSokolov)\ , [@AshesITR](https://github.com/AshesITR)\ , [@asmlgkj](https://github.com/asmlgkj)\ , [@aubryvetepi](https://github.com/aubryvetepi)\ , [@balwierz](https://github.com/balwierz)\ , [@bastianilso](https://github.com/bastianilso)\ , [@batpigandme](https://github.com/batpigandme)\ , [@bebersb](https://github.com/bebersb)\ , [@behrman](https://github.com/behrman)\ , [@benjaminschwetz](https://github.com/benjaminschwetz)\ , [@billdenney](https://github.com/billdenney)\ , [@Breza](https://github.com/Breza)\ , [@brunj7](https://github.com/brunj7)\ , [@BrunoGrandePhD](https://github.com/BrunoGrandePhD)\ , [@CGMossa](https://github.com/CGMossa)\ , [@cgoo4](https://github.com/cgoo4)\ , [@chsafouane](https://github.com/chsafouane)\ , [@chumbleycode](https://github.com/chumbleycode)\ , [@ColinFay](https://github.com/ColinFay)\ , [@CorradoLanera](https://github.com/CorradoLanera)\ , [@CPRyan](https://github.com/CPRyan)\ , [@czeildi](https://github.com/czeildi)\ , [@dan-reznik](https://github.com/dan-reznik)\ , [@DanChaltiel](https://github.com/DanChaltiel)\ , [@datawookie](https://github.com/datawookie)\ , [@dave-lovell](https://github.com/dave-lovell)\ , [@davidsjoberg](https://github.com/davidsjoberg)\ , [@DavisVaughan](https://github.com/DavisVaughan)\ , [@deann88](https://github.com/deann88)\ , [@dfalbel](https://github.com/dfalbel)\ , [@dhslone](https://github.com/dhslone)\ , [@dlependorf](https://github.com/dlependorf)\ , [@dllazarov](https://github.com/dllazarov)\ , [@dpprdan](https://github.com/dpprdan)\ , [@dracodoc](https://github.com/dracodoc)\ , [@echasnovski](https://github.com/echasnovski)\ , [@edo91](https://github.com/edo91)\ , [@edoardo-oliveri-sdg](https://github.com/edoardo-oliveri-sdg)\ , [@erictleung](https://github.com/erictleung)\ , [@eyayaw](https://github.com/eyayaw)\ , [@felixhell2004](https://github.com/felixhell2004)\ , [@florianm](https://github.com/florianm)\ , [@florisvdh](https://github.com/florisvdh)\ , [@flying-sheep](https://github.com/flying-sheep)\ , [@fpinter](https://github.com/fpinter)\ , [@frankzhang21](https://github.com/frankzhang21)\ , [@gaborcsardi](https://github.com/gaborcsardi)\ , [@GarrettMooney](https://github.com/GarrettMooney)\ , [@gdurif](https://github.com/gdurif)\ , [@ge-li](https://github.com/ge-li)\ , [@ggrothendieck](https://github.com/ggrothendieck)\ , [@grayskripko](https://github.com/grayskripko)\ , [@gregleleu](https://github.com/gregleleu)\ , [@gregorp](https://github.com/gregorp)\ , [@hadley](https://github.com/hadley)\ , [@hendrikvanb](https://github.com/hendrikvanb)\ , [@holgerbrandl](https://github.com/holgerbrandl)\ , [@hriebl](https://github.com/hriebl)\ , [@hsloot](https://github.com/hsloot)\ , [@huftis](https://github.com/huftis)\ , [@iago-pssjd](https://github.com/iago-pssjd)\ , [@iamnicogomez](https://github.com/iamnicogomez)\ , [@IndrajeetPatil](https://github.com/IndrajeetPatil)\ , [@irudnyts](https://github.com/irudnyts)\ , [@izahn](https://github.com/izahn)\ , [@jameslairdsmith](https://github.com/jameslairdsmith)\ , [@jedwards24](https://github.com/jedwards24)\ , [@jemus42](https://github.com/jemus42)\ , [@jennybc](https://github.com/jennybc)\ , [@jhrcook](https://github.com/jhrcook)\ , [@jimhester](https://github.com/jimhester)\ , [@jimjam-slam](https://github.com/jimjam-slam)\ , [@jnolis](https://github.com/jnolis)\ , [@joelgombin](https://github.com/joelgombin)\ , [@jonathan-g](https://github.com/jonathan-g)\ , [@jpmarindiaz](https://github.com/jpmarindiaz)\ , [@jxu](https://github.com/jxu)\ , [@jzadra](https://github.com/jzadra)\ , [@karchjd](https://github.com/karchjd)\ , [@karjamatti](https://github.com/karjamatti)\ , [@kbzsl](https://github.com/kbzsl)\ , [@krlmlr](https://github.com/krlmlr)\ , [@lahvak](https://github.com/lahvak)\ , [@lambdamoses](https://github.com/lambdamoses)\ , [@lasuk](https://github.com/lasuk)\ , [@lionel-](https://github.com/lionel-)\ , [@lorenzwalthert](https://github.com/lorenzwalthert)\ , [@LukasWallrich](https://github.com/LukasWallrich)\ , [@LukaszDerylo](https://github.com/LukaszDerylo)\ , [@malcolmbarrett](https://github.com/malcolmbarrett)\ , [@MarceloRTonon](https://github.com/MarceloRTonon)\ , [@mattwarkentin](https://github.com/mattwarkentin)\ , [@maxheld83](https://github.com/maxheld83)\ , [@Maximilian-Stefan-Ernst](https://github.com/Maximilian-Stefan-Ernst)\ , [@mccroweyclinton-EPA](https://github.com/mccroweyclinton-EPA)\ , [@medewitt](https://github.com/medewitt)\ , [@meowcat](https://github.com/meowcat)\ , [@mgirlich](https://github.com/mgirlich)\ , [@mine-cetinkaya-rundel](https://github.com/mine-cetinkaya-rundel)\ , [@mitchelloharawild](https://github.com/mitchelloharawild)\ , [@mkoohafkan](https://github.com/mkoohafkan)\ , [@mlane3](https://github.com/mlane3)\ , [@mmuurr](https://github.com/mmuurr)\ , [@moodymudskipper](https://github.com/moodymudskipper)\ , [@mpettis](https://github.com/mpettis)\ , [@nealrichardson](https://github.com/nealrichardson)\ , [@Nelson-Gon](https://github.com/Nelson-Gon)\ , [@neuwirthe](https://github.com/neuwirthe)\ , [@njtierney](https://github.com/njtierney)\ , [@oduilln](https://github.com/oduilln)\ , [@papageorgiou](https://github.com/papageorgiou)\ , [@pat-s](https://github.com/pat-s)\ , [@paulponcet](https://github.com/paulponcet)\ , [@petyaracz](https://github.com/petyaracz)\ , [@phargarten2](https://github.com/phargarten2)\ , [@philiporlando](https://github.com/philiporlando)\ , [@q-w-a](https://github.com/q-w-a)\ , [@QuLogic](https://github.com/QuLogic)\ , [@ramiromagno](https://github.com/ramiromagno)\ , [@rcorty](https://github.com/rcorty)\ , [@reisner](https://github.com/reisner)\ , [@Rekyt](https://github.com/Rekyt)\ , [@roboes](https://github.com/roboes)\ , [@romainfrancois](https://github.com/romainfrancois)\ , [@rorynolan](https://github.com/rorynolan)\ , [@salim-b](https://github.com/salim-b)\ , [@sar8421](https://github.com/sar8421)\ , [@ScoobyQ](https://github.com/ScoobyQ)\ , [@sda030](https://github.com/sda030)\ , [@sgschreiber](https://github.com/sgschreiber)\ , [@sheffe](https://github.com/sheffe)\ , [@Shians](https://github.com/Shians)\ , [@ShixiangWang](https://github.com/ShixiangWang)\ , [@shosaco](https://github.com/shosaco)\ , [@siavash-babaei](https://github.com/siavash-babaei)\ , [@stephenashton-dhsc](https://github.com/stephenashton-dhsc)\ , [@stschiff](https://github.com/stschiff)\ , [@surdina](https://github.com/surdina)\ , [@tdawry](https://github.com/tdawry)\ , [@thebioengineer](https://github.com/thebioengineer)\ , [@TimTaylor](https://github.com/TimTaylor)\ , [@TimTeaFan](https://github.com/TimTeaFan)\ , [@tomjemmett](https://github.com/tomjemmett)\ , [@torbjorn](https://github.com/torbjorn)\ , [@tvatter](https://github.com/tvatter)\ , [@TylerGrantSmith](https://github.com/TylerGrantSmith)\ , [@vorpalvorpal](https://github.com/vorpalvorpal)\ , [@vspinu](https://github.com/vspinu)\ , [@wch](https://github.com/wch)\ , [@werkstattcodes](https://github.com/werkstattcodes)\ , [@williamlai2](https://github.com/williamlai2)\ , [@yogat3ch](https://github.com/yogat3ch)\ , [@yutannihilation](https://github.com/yutannihilation)\ , and [@zeehio](https://github.com/zeehio)\ .\ \ * * *\ \ 1. But if they used the tidyverse coercion rules. [↩︎](#fnref:1)\ \ \ Contents\ \ The tidyverse is proudly supported by[](https://posit.co/)\ \ [Privacy policy](https://www.tidyverse.org/google_privacy_policy)\ [](https://github.com/tidyverse)\ [](https://twitter.com/hashtag/rstats) --- # ggplot2 3.4.0 [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) ggplot2 3.4.0 ============= ![](/blog/2022/11/ggplot2-3-4-0/thumbnail-wd.jpg) Photo by [Thomas Lin Pedersen](https://data-imaginist.com/art)   2022/11/07   [graphics](/tags/graphics/) , [ggplot2](/tags/ggplot2/)   Thomas Lin Pedersen We’re so happy to announce the release of [ggplot2](https://ggplot2.tidyverse.org) 3.4.0 on CRAN. ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details. The new version can be installed from CRAN using `install.packages("ggplot2")`. This release is not full of exciting new features. Instead we have focused on the internals, tightening up of the API, and improving the messaging, especially when it comes to errors and warnings. While the release also contains a few new features these other aspects are the stars of this release. You can see a full list of changes in the [release notes](https://ggplot2.tidyverse.org/news/index.html) library(ggplot2) library(patchwork) library(dplyr) Hello `linewidth`[](#hello-linewidth) -------------------------------------- Arguably the biggest user.visible change in this release is the introduction of a new fundamental aesthetic. From this release on, `linewidth` will take over sizing of the width of lines—something that was earlier handled by `size`. The reason for this change is that prior to this release `size` was used for two related, but different, properties: the size of points (and glyphs) and the width of lines. Since one is area based and one is length based they fundamentally needs different scaling and the default size scale has always catered to area sizing, using a square root transform. This conflation has also made it hard for composite geoms like [`geom_pointrange()`](https://ggplot2.tidyverse.org/reference/geom_linerange.html) to control the line width and point size separately. There is not much to discuss when it comes to how to use this “feature”, as it is a matter of switching out `size` with `linewidth` whenever you target stroke sizing: ggplot(airquality) + geom_line(aes(Day, Temp, linewidth = Month, group = Month)) + scale_linewidth(range = c(0.5, 3)) ![](figs/unnamed-chunk-1-1.png) Now, changing such a fundamental thing when a package is as old and widely used as ggplot2 is no small undertaking, and I wish it had been done earlier, but better late than never. We have gone to great lengths to ensure that old code continues to work. For the most part using size will continue to behave like before: ggplot(airquality) + geom_line(aes(Day, Temp, size = Month, group = Month)) + scale_size(range = c(0.5, 3)) #> Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0. #> ℹ Please use `linewidth` instead. ![](figs/unnamed-chunk-2-1.png) As you can see you get the expected plot but also gets a deprecation warning asking you to update your code. Comparing the two legends we can also see the discrepancy in scaling that we discussed above, showing a much more even progression with `linewidth`. All of this should work with all the geoms provided by ggplot2 (and we have described [a clear upgrade path for extension developers to adopt this](https://www.tidyverse.org/blog/2022/08/ggplot2-3-4-0-size-to-linewidth/) ), except for a few instances where `size` remains a valid aesthetic for the geom. In these cases you will not get a deprecation warning and your output may change in look when running old code. The two geoms this concerns are [`geom_pointrange()`](https://ggplot2.tidyverse.org/reference/geom_linerange.html) and [`geom_sf()`](https://ggplot2.tidyverse.org/reference/ggsf.html) which both continues to use `size` to scale points. Comparing the output from e.g.  [`geom_pointrange()`](https://ggplot2.tidyverse.org/reference/geom_linerange.html) we can see how using `size` now only targets the point and not the line: ggplot(airquality) + geom_pointrange(aes(x = factor(Month), y = Temp), stat = "summary", size = 2) #> No summary function supplied, defaulting to `mean_se()` ![](figs/unnamed-chunk-3-1.png) We recognize that introducing silent visual changes like this is not optimal but we weighted both sides and decided that it was better in the long run to rip the band-aid off and commit fully to the `linewidth` change in one release. The switch to `linewidth` goes beyond aesthetics and should target every part of the API that have used `size` to target line width. This is mostly present in theming where [`element_rect()`](https://ggplot2.tidyverse.org/reference/element.html) and [`element_line()`](https://ggplot2.tidyverse.org/reference/element.html) now uses `linewidth` as argument instead of `size`. As above a deprecation warning will inform you of this change: ggplot(mtcars) + geom_point(aes(x = mpg, y = disp)) + theme(panel.grid = element_line(linewidth = 0.2)) ![](figs/unnamed-chunk-4-1.png) We have done our best to ensure that it is easy for our extension developers to follow the path laid out by ggplot2 when it comes to embracing the new aesthetic, but you will probably experience a period of discrepancy between some of your favorite extensions and ggplot2. I have full confidence that our amazing extension developers will adapt quickly so that period will probably be short. ### On the topic of line width[](#on-the-topic-of-line-width) We have made a few other internal changes when it comes to line widths. The biggest of these are perhaps a new default for polygon line width in [`geom_sf()`](https://ggplot2.tidyverse.org/reference/ggsf.html) . The change came about as we already had induced visual changes to old code due to the `linewidth` aesthetic introduction and based on feedback from the spatial community we saw that `size` was most often used to thin the polygon borders. The new default is 0.2 (down from 0.5) and hopefully strikes a nice balance: nc <- sf::st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE) p1 <- ggplot(nc) + geom_sf(linewidth = 0.5) + ggtitle("Old default") p2 <- ggplot(nc) + geom_sf() + ggtitle("New default") p1/p2 ![](figs/unnamed-chunk-5-1.png) More minor is a small fix we did to [`guide_colorbar()`](https://ggplot2.tidyverse.org/reference/guide_colourbar.html) where it was brought to our attention that the `ticks.linewidth` and `frame.linewidth` weren’t given in the same unit as every other line width in ggplot2. This has been corrected and the default has been adjusted to retain the same look but if you have given these specifically in your code you are likely to notice a visual change. Other breaking changes[](#other-breaking-changes) -------------------------------------------------- In the grab-bag of breaking changes we have now formally deprecated [`qplot()`](https://ggplot2.tidyverse.org/reference/qplot.html) . It will continue to work as always but will be a bit noisy about it. Don’t expect the function to disappear, but the deprecation signals that we don’t intend to do further work on [`qplot()`](https://ggplot2.tidyverse.org/reference/qplot.html) to keep it current with new features etc. In the same vein, [`stat()`](https://ggplot2.tidyverse.org/reference/aes_eval.html) and `..var..` for marking aesthetics from stats are also formally deprecated in favor of [`after_stat()`](https://ggplot2.tidyverse.org/reference/aes_eval.html) . Again, the result is that using these old APIs will be noisy but still work. On the topic of [`after_stat()`](https://ggplot2.tidyverse.org/reference/aes_eval.html) , the values and computations inside of it now use the un-transformed variables rather than the transformed ones. This is a bit esoteric and only applies to aesthetics that have had a scale transformation applied to them, so you may never notice. Lastly, we have made a switch to using [`rlang::hash()`](https://rlang.r-lib.org/reference/hash.html) instead of [`digest::digest()`](https://rdrr.io/pkg/digest/man/digest.html) which may result in the automatic ordering of legends changing, again a pretty minor change. If you care about the ordering of the legends you can always take control of it using the `order` argument inside the different `guide_*()` constructors. Better errors[](#better-errors) -------------------------------- One of the most substantial changes in usability in this release is a complete rewrite of the errors and warnings. This goes deeper than changing wordings as the messaging is now based on the signal handling in the [cli](https://cli.r-lib.org) package that provides rich text formatting and better ways to guide the user to a resolution. Consider the following easy to make mistake of using the pipe instead of `+`: ggplot(mtcars) |> geom_point(aes(mpg, disp)) #> Error in `geom_point()`: #> ! `mapping` must be created by `aes()` #> ℹ Did you use `%>%` or `|>` instead of `+`? As can be seen, the error now clearly states where it is happening, then tells you what is wrong, and lastly gives you a hint at what might be the solution. However, this is not all. One of the biggest issues with error reporting in ggplot2 is that most code is evaluated during rendering, not when the API calls are made. Because of this it has been difficult to link a user error in a geom specification to the actual error message that arises. This could send the user on a treasure hunt to identify what to change in order to fix the code. With the changes in 3.4.0 we are now much better at directing the user to the right place in their code when errors in the rendering happens: huron <- data.frame(year = 1875:1972, level = as.vector(LakeHuron)) ggplot(huron) + geom_line(aes(year, level)) + geom_ribbon(aes(year, xmin = level - 5, xmax = level + 5)) #> Error in `geom_ribbon()`: #> ! Problem while setting up geom. #> ℹ Error occurred in the 2nd layer. #> Caused by error in `compute_geom_1()` at ggplot2/R/ggproto.r:182:16: #> ! `geom_ribbon()` requires the following missing aesthetics: ymin and #> ymax or y We can see that the error message correctly identifies the geom responsible for the layer, communicates during what part of the rendering it happened during, and points to the index of the layer in the case that multiple layers from the same geom have been used. Lastly it shows the original error that can help you with solving the issue. Hopefully the changes goes a long way to make ggplot2 even more welcoming to new and seasoned users alike. However, this effort is never done and we continue to appreciate issues in the github repository pointing out unhelpful errors or warnings that arises so that we may improve it further. vctrs inside[](#vctrs-inside) ------------------------------ The last part of the large housekeeping changes in this release is that ggplot2 finally embraces [vctrs](https://vctrs.r-lib.org) and uses it’s functions internally primarily for binding data together. Apart from a nice bump in rendering speed it also means that we now better support data types built upon vctrs and subscribe to the more well-defined coercion rules that it provides. The last point is a double edged sword though, as your code may contain a diverse mix of data types in different layers that worked before but doesn’t align with the strictness of vctrs. While we have gone to lengths to ensure that your code still works you will begin to see deprecation notices if you e.g. factor on a variable that is incompatible across layers: labels <- data.frame( label = paste("gear", 3:5), gear = as.character(3:5), x = 100, y = 11 ) ggplot(mtcars) + geom_point(aes(disp, mpg)) + geom_text(aes(x, y, label = label), labels, hjust = "left") + facet_wrap(~gear) #> Warning: Combining variables of class and was deprecated in #> ggplot2 3.4.0. #> ℹ Please ensure your variables are compatible before plotting (location: #> `combine_vars()`) ![](figs/unnamed-chunk-8-1.png) While this may seem like an unnecessary annoyance we hope that you’ll learn to appreciate that this strictness can save you from silent bugs where you end up combining variables that are basically incompatible. New features[](#new-features) ------------------------------ While most of the focus has been on internal housekeeping in this release a few new features has also crept in, courtesy of our amazing contributors from the community: ### Stacking non-aligned data[](#stacking-non-aligned-data) [`position_stack()`](https://ggplot2.tidyverse.org/reference/position_stack.html) has always required that groups share a common x-value to be stacked. The nature of most time series data etc. makes it so that this is often the case, but not always. We have now introduced a [`stat_align()`](https://ggplot2.tidyverse.org/reference/geom_ribbon.html) that takes care of interpolating y-values in each group at every unique x-value in the data so that they can be stacked. This stat is now the default for [`geom_area()`](https://ggplot2.tidyverse.org/reference/geom_ribbon.html) : df <- tibble::tribble( ~g, ~x, ~y, "a", 1, 2, "a", 3, 5, "a", 5, 1, "b", 2, 0, "b", 4, 6, "b", 6, 7 ) p1 <- ggplot(df, aes(x, y, fill = g)) + geom_area(stat = "identity", alpha = 0.5) + ggtitle("stat_identity()") p2 <- ggplot(df, aes(x, y, fill = g)) + geom_area(alpha = 0.5) + ggtitle("stat_align()") (p1 | p2) & theme(legend.position = "none") ![](figs/unnamed-chunk-9-1.png) ### Bounded density estimation[](#bounded-density-estimation) [`geom_density()`](https://ggplot2.tidyverse.org/reference/geom_density.html) have gained a `bounds` argument allowing you to perform density estimation with bound correction. This can leads to might better edge estimates when bounds are known for a sample: data <- data.frame(x = rexp(100)) ggplot(data, aes(x)) + geom_density(aes(colour = "unbounded"), key_glyph = "path") + geom_density(aes(colour = "bounded"), bounds = c(0, Inf), key_glyph = "path") + stat_function(aes(colour = "true distribution"), fun = dexp) + scale_colour_manual( name = NULL, values = c("black", "firebrick", "forestgreen"), breaks = c("true distribution", "unbounded", "bounded") ) ![](figs/unnamed-chunk-10-1.png) ### No clipping in facet strips[](#no-clipping-in-facet-strips) It is now possible to turn clipping in the facet strips off. For the most part the default works fine but in certain situations you’d like the strip text or the border to be seen in full. The new feature is a theme setting: p <- ggplot(diamonds) + geom_bar(aes(y = color)) + facet_wrap(~ cut) + theme_minimal() + theme( strip.background = element_rect("grey90", colour = "grey90", linewidth = 1), axis.line.y = element_line(linewidth = 1) ) p ![](figs/unnamed-chunk-11-1.png) In the (a bit contrived) theme above we see a jarring step between the strip background and the axis line because the border of the strip is clipped to the extent of the strip. We can fix this by turning off clipping: p + theme(strip.clip = "off") ![](figs/unnamed-chunk-12-1.png) ### Justification in `geom_bar()`/`geom_col()`[](#justification-in-geom_bargeom_col) You can now specify how the bars in [`geom_bar()`](https://ggplot2.tidyverse.org/reference/geom_bar.html) should be justified with respect to the position on the axis they are tied to: mtcars_centered <- mutate(mtcars, justification = "centered") mtcars_left <- mutate(mtcars, justification = "left aligned") ggplot(mapping = aes(x = gear)) + geom_bar(data = mtcars_centered) + geom_bar(data = mtcars_left, just = 0) + facet_wrap(~justification, ncol = 1) ![](figs/unnamed-chunk-13-1.png) It goes without saying that you should only do this for good reasons because it goes against how people in general expect bar plots to behave, but for certain layout needs it can be a boon. Acknowledgements[](#acknowledgements) -------------------------------------- As always, this release could not be possible without contributions from our amazing community. A huge thanks goes out to everyone who has helped made ggplot2 3.4.0 a reality: [@92amartins](https://github.com/92amartins) , [@acircleda](https://github.com/acircleda) , [@AlgaeKat](https://github.com/AlgaeKat) , [@andreaskuepfer](https://github.com/andreaskuepfer) , [@angleik](https://github.com/angleik) , [@aphalo](https://github.com/aphalo) , [@artuurC](https://github.com/artuurC) , [@asolisc](https://github.com/asolisc) , [@baderstine](https://github.com/baderstine) , [@basille](https://github.com/basille) , [@bergsmat](https://github.com/bergsmat) , [@bersbersbers](https://github.com/bersbersbers) , [@billdenney](https://github.com/billdenney) , [@brianmsm](https://github.com/brianmsm) , [@brunomioto](https://github.com/brunomioto) , [@bwiernik](https://github.com/bwiernik) , [@capnrefsmmat](https://github.com/capnrefsmmat) , [@clauswilke](https://github.com/clauswilke) , [@cmartin](https://github.com/cmartin) , [@ConchuirohAodha](https://github.com/ConchuirohAodha) , [@corybrunson](https://github.com/corybrunson) , [@DanChaltiel](https://github.com/DanChaltiel) , [@DarioS](https://github.com/DarioS) , [@Darxor](https://github.com/Darxor) , [@davidchall](https://github.com/davidchall) , [@davidhodge931](https://github.com/davidhodge931) , [@dhrhzz](https://github.com/dhrhzz) , [@DiegoJArg](https://github.com/DiegoJArg) , [@DISOhda](https://github.com/DISOhda) , [@drtoche](https://github.com/drtoche) , [@Enterprise-J](https://github.com/Enterprise-J) , [@ewallace](https://github.com/ewallace) , [@gbrlrgrs](https://github.com/gbrlrgrs) , [@ggrothendieck](https://github.com/ggrothendieck) , [@GregorDall](https://github.com/GregorDall) , [@hadley](https://github.com/hadley) , [@henningpohl](https://github.com/henningpohl) , [@Hugh-Mungo](https://github.com/Hugh-Mungo) , [@IndrajeetPatil](https://github.com/IndrajeetPatil) , [@JacobElder](https://github.com/JacobElder) , [@jarauh](https://github.com/jarauh) , [@javlon](https://github.com/javlon) , [@jdonland](https://github.com/jdonland) , [@jessexknight](https://github.com/jessexknight) , [@jfunction](https://github.com/jfunction) , [@JobNmadu](https://github.com/JobNmadu) , [@JoFAM](https://github.com/JoFAM) , [@jooyoungseo](https://github.com/jooyoungseo) , [@jpquast](https://github.com/jpquast) , [@jtlandis](https://github.com/jtlandis) , [@junjunlab](https://github.com/junjunlab) , [@jwhendy](https://github.com/jwhendy) , [@kapsner](https://github.com/kapsner) , [@KasperThystrup](https://github.com/KasperThystrup) , [@kongdd](https://github.com/kongdd) , [@LarryVincent](https://github.com/LarryVincent) , [@leonjessen](https://github.com/leonjessen) , [@Lisamrshhsr](https://github.com/Lisamrshhsr) , [@llrs](https://github.com/llrs) , [@LuisLauM](https://github.com/LuisLauM) , [@lynn242](https://github.com/lynn242) , [@makrez](https://github.com/makrez) , [@MichaelChirico](https://github.com/MichaelChirico) , [@michaelgrund](https://github.com/michaelgrund) , [@mikeroswell](https://github.com/mikeroswell) , [@mjsmith037](https://github.com/mjsmith037) , [@moodymudskipper](https://github.com/moodymudskipper) , [@mvanaman](https://github.com/mvanaman) , [@netique](https://github.com/netique) , [@nfancy](https://github.com/nfancy) , [@ngreifer](https://github.com/ngreifer) , [@nkehrein](https://github.com/nkehrein) , [@olobiolo](https://github.com/olobiolo) , [@orgadish](https://github.com/orgadish) , [@pachadotdev](https://github.com/pachadotdev) , [@padpadpadpad](https://github.com/padpadpadpad) , [@paupaiz](https://github.com/paupaiz) , [@ProfessorPeregrine](https://github.com/ProfessorPeregrine) , [@PursuitOfDataScience](https://github.com/PursuitOfDataScience) , [@rikudoukarthik](https://github.com/rikudoukarthik) , [@rjake](https://github.com/rjake) , [@rressler](https://github.com/rressler) , [@SarenT](https://github.com/SarenT) , [@Sebas256](https://github.com/Sebas256) , [@shenzhenzth](https://github.com/shenzhenzth) , [@skyroam](https://github.com/skyroam) , [@stargorg](https://github.com/stargorg) , [@stefanoborini](https://github.com/stefanoborini) , [@steveharoz](https://github.com/steveharoz) , [@stragu](https://github.com/stragu) , [@szimmer](https://github.com/szimmer) , [@tamas-ferenci](https://github.com/tamas-ferenci) , [@teunbrand](https://github.com/teunbrand) , [@tfjaeger](https://github.com/tfjaeger) , [@thomasp85](https://github.com/thomasp85) , [@thoolihan](https://github.com/thoolihan) , [@tjebo](https://github.com/tjebo) , [@topepo](https://github.com/topepo) , [@trevorld](https://github.com/trevorld) , [@tungttnguyen](https://github.com/tungttnguyen) , [@twest820](https://github.com/twest820) , [@waynerroper](https://github.com/waynerroper) , [@willgearty](https://github.com/willgearty) , [@wmacnair](https://github.com/wmacnair) , [@wurli](https://github.com/wurli) , [@yutannihilation](https://github.com/yutannihilation) , and [@zeehio](https://github.com/zeehio) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # tidyselect [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Tidyselect [![](/blog/2022/10/tidyselect-1-2-0/thumbnail-sq.jpg)](/blog/2022/10/tidyselect-1-2-0/) [tidyselect 1.2.0](/blog/2022/10/tidyselect-1-2-0/) [package](/categories/package) Lionel Henry and Hadley Wickham tidyselect 1.2.0 hit CRAN last week and includes a few updates to the syntax of selections in tidyverse functions. [Read more ...](/blog/2022/10/tidyselect-1-2-0/) 2022/10/18 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # Q3 2022 tidymodels digest [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Q3 2022 tidymodels digest ========================= ![](/blog/2022/10/tidymodels-2022-q3/thumbnail-wd.jpg) Photo by [Simon Spieske](https://unsplash.com/photos/PyDaL4PcLoQ)   2022/10/19   [tidymodels](/tags/tidymodels/) , [recipes](/tags/recipes/) , [agua](/tags/agua/) , [h2o](/tags/h2o/)   Max Kuhn The [tidymodels](https://www.tidymodels.org/) framework is a collection of R packages for modeling and machine learning using tidyverse principles. Since the beginning of 2021, we have been publishing [quarterly updates](https://www.tidyverse.org/categories/roundup/) here on the tidyverse blog summarizing what’s new in the tidymodels ecosystem. The purpose of these regular posts is to share useful new features and any updates you may have missed. You can check out the [`tidymodels` tag](https://www.tidyverse.org/tags/tidymodels/) to find all tidymodels blog posts here, including our roundup posts as well as those that are more focused, like these from the past month or so: * [Improvements to model specification checking in tidymodels](https://www.tidyverse.org/blog/2022/10/parsnip-checking-1-0-2/) * [brulee 0.2.0](https://www.tidyverse.org/blog/2022/09/brulee-0-2-0/) * [Announcing bundle](https://www.tidyverse.org/blog/2022/09/bundle-0-1-0/) * [censored 0.1.0](https://www.tidyverse.org/blog/2022/08/censored-0-1-0/) * [rsample 1.1.0](https://www.tidyverse.org/blog/2022/08/rsample-1-1-0/) Since [our last roundup post](https://www.tidyverse.org/blog/2022/07/tidymodels-2022-q2/) , there have been CRAN releases of 22 tidymodels packages. Here are links to their NEWS files: * agua [(0.1.0)](https://agua.tidymodels.org/news/index.html) * applicable [(0.1.0)](https://github.com/tidymodels/applicable/blob/develop/NEWS.md) * bonsai [(0.2.0)](https://bonsai.tidymodels.org/news/index.html) * broom [(1.0.1)](https://broom.tidymodels.org/news/index.html) * brulee [(0.2.0)](https://brulee.tidymodels.org/news/index.html) * butcher [(0.3.0)](https://butcher.tidymodels.org/news/index.html) * censored [(0.1.1)](https://censored.tidymodels.org/news/index.html) * corrr [(0.4.4)](https://corrr.tidymodels.org/news/index.html) * finetune [(1.0.1)](https://finetune.tidymodels.org/news/index.html) * modeldata [(1.0.1)](https://modeldata.tidymodels.org/news/index.html) * modeldb [(0.2.3)](https://modeldb.tidymodels.org/news/index.html) * parsnip [(1.0.2)](https://parsnip.tidymodels.org/news/index.html) * plsmod [(1.0.0)](https://plsmod.tidymodels.org/news/index.html) * poissonreg [(1.0.1)](https://poissonreg.tidymodels.org/news/index.html) * probably [(0.1.0)](https://probably.tidymodels.org/news/index.html) * recipes [(1.0.2)](https://recipes.tidymodels.org/news/index.html) * rsample [(1.1.0)](https://rsample.tidymodels.org/news/index.html) * spatialsample [(0.2.1)](https://spatialsample.tidymodels.org/news/index.html) * textrecipes [(1.0.1)](https://textrecipes.tidymodels.org/news/index.html) * tune [(1.0.1)](https://tune.tidymodels.org/news/index.html) * workflows [(1.1.0)](https://workflows.tidymodels.org/news/index.html) * yardstick [(1.1.0)](https://yardstick.tidymodels.org/news/index.html) We’ll highlight two specific upgrades: one for agua and another for recipes. A big upgrade for agua[](#a-big-upgrade-for-agua) -------------------------------------------------- With version 3.38.0.1 of the [h2o](https://cran.r-project.org/package=h2o) package, agua can now tune h2o models as if they were any other type of model engine. [h2o](https://h2o.ai) has an excellent server-based computational engine for fitting a variety of different machine learning and statistical models. The h2o server can run locally or on some external high performance computing server. The downside is that it is light on tools for feature engineering and interactive data analysis. Using h2o with tidymodels enables users to leverage the benefits of packages like recipes along with fast, server-based parallel processing. While the syntax for model fitting and tuning are the same as any other non-h2o model, there are different ways to parallelize the work: * The h2o server has the ability to internally parallelize individual model computations. For example, when fitting trees, the search for the best split can be done using multiple threads. The number of threads that each model should be used is set with [h2o.init(nthreads)](https://docs.h2o.ai/h2o/latest-stable/h2o-r/docs/reference/h2o.init.html) . The default (`-1`) is to use all CPUs on the host. * When using grid search, [h2o.grid(parallelism)](https://docs.h2o.ai/h2o/latest-stable/h2o-r/docs/reference/h2o.grid.html) determines how many models the h2o server should process at the same time. The default (`1`) constrains the server to run the models sequentially. * R has external parallelization tools (such as the foreach and future packages) that can start new R processes to simultaneously do work. This would run many models in parallel. For h2o, this determines how many models the agua package could send to the server at once. This does not appear to be constrained by the `parallelism` argument to `h2o.grid()`. With h2o and tidymodels, you should probably **use h2o’s parallelization**. Using multiple approaches _can_ work but only for some technologies. It’s still [pretty complicated](https://github.com/topepo/agua-h2o-benchmark) but we are working on un-complicating it. To set up h2o parallelization, there is a new control argument called `backend_options`. If you were doing a grid search, you first define how many threads the h2o server should use: library(tidymodels) library(agua) library(finetune) h2o_thread_spec <- agua_backend_options(parallelism = 10) Then, pass the output to any of the existing control functions: grid_ctrl <- control_grid(backend_options = h2o_thread_spec) Now h2o can parallel process 10 models at once. Here is an example using a simulated data set with a numeric outcome: library(tidymodels) library(agua) library(finetune) # Simulate the data n_train <- 200 set.seed(6147) sim_dat <- sim_regression(n_train) # Resample using 10-fold cross-validation set.seed(91) sim_rs <- vfold_cv(sim_dat) We’ll use grid search to tune a boosted tree: boost_spec <- boost_tree( trees = tune(), min_n = tune(), tree_depth = tune(), learn_rate = tune(), loss_reduction = tune() ) %>% set_engine("h2o") %>% set_mode("regression") Now, let’s parallel process our computations. # Start the h2o server h2o::h2o.init() # Multi-thread the model fits h2o_thread_spec <- agua_backend_options(parallelism = 10) grid_ctrl <- control_grid(backend_options = h2o_thread_spec) We’ll evaluate a very small grid at first: set.seed(7616) grid_res <- boost_spec %>% tune_grid(outcome ~ ., resamples = sim_rs, grid = 10, control = grid_ctrl) show_best(grid_res, metric = "rmse") %>% select(-.config, -.metric, -.estimator) #> # A tibble: 5 × 8 #> trees min_n tree_depth learn_rate loss_reduction mean n std_err #> #> 1 1954 17 13 0.0318 6.08e-8 13.1 10 0.828 #> 2 184 25 4 0.00000000164 6.56e-1 15.7 10 1.03 #> 3 1068 10 8 0.0000409 1.19e+1 17.4 10 1.08 #> 4 1500 37 10 0.0000108 9.97e-9 18.3 10 1.03 #> 5 985 18 7 0.0000000454 1.84e-3 18.4 10 1.04 autoplot(grid_res, metric = "rmse") ![](figs/grid-plot-1.svg) It was a small grid and most of the configurations were not especially good. We can further optimize the results by applying simulated annealing search to the best grid results. sa_ctrl <- control_sim_anneal(backend_options = h2o_thread_spec) set.seed(4) sa_res <- boost_spec %>% tune_sim_anneal( outcome ~ ., resamples = sim_rs, initial = grid_res, iter = 25, control = sa_ctrl ) #> Optimizing rmse #> Initial best: 13.06400 #> 1 ♥ new best rmse=12.688 (+/-0.7899) #> 2 ◯ accept suboptimal rmse=12.849 (+/-0.8304) #> 3 ◯ accept suboptimal rmse=13.129 (+/-0.8266) #> 4 ◯ accept suboptimal rmse=13.678 (+/-0.9544) #> 5 + better suboptimal rmse=13.433 (+/-0.792) #> 6 + better suboptimal rmse=12.99 (+/-0.9031) #> 7 ─ discard suboptimal rmse=16.531 (+/-1.027) #> 8 ─ discard suboptimal rmse=13.522 (+/-0.9802) #> 9 ✖ restart from best rmse=13.097 (+/-0.8109) #> 10 ♥ new best rmse=12.66 (+/-0.8028) #> 11 ◯ accept suboptimal rmse=13.116 (+/-0.8135) #> 12 + better suboptimal rmse=12.714 (+/-0.7747) #> 13 ─ discard suboptimal rmse=13.074 (+/-0.6598) #> 14 ─ discard suboptimal rmse=14.489 (+/-1.028) #> 15 ◯ accept suboptimal rmse=12.715 (+/-0.8043) #> 16 ─ discard suboptimal rmse=13.788 (+/-1.027) #> 17 ─ discard suboptimal rmse=13.057 (+/-0.7716) #> 18 ✖ restart from best rmse=13.064 (+/-0.7095) #> 19 ♥ new best rmse=12.645 (+/-0.7706) #> 20 ◯ accept suboptimal rmse=12.7 (+/-0.821) #> 21 ─ discard suboptimal rmse=13.018 (+/-0.8047) #> 22 ─ discard suboptimal rmse=14.812 (+/-1.017) #> 23 ─ discard suboptimal rmse=13.098 (+/-0.921) #> 24 ◯ accept suboptimal rmse=12.708 (+/-0.7538) #> 25 ◯ accept suboptimal rmse=13.054 (+/-0.9046) Again, h2o is doing all of the computational work for fitting models and tidymodels is proposing new parameter configurations. One other nice feature of the new agua release is the h2o engine for the [`auto_ml()`](https://parsnip.tidymodels.org/reference/auto_ml.html) model. This builds a stacked ensemble on a set of different models (not unlike our [stacks](https://stacks.tidymodels.org) package but with far less code). There is a great worked example [on the agua website](https://agua.tidymodels.org/articles/auto_ml.html) so make sure to check this out! More spline recipe steps[](#more-spline-recipe-steps) ------------------------------------------------------ Spline techniques allow linear models to produce nonlinear model curves. These are called [basis expansion methods](https://bookdown.org/max/FES/numeric-one-to-many.html#numeric-basis-functions) since they take a single numeric predictor and make additional nonlinear feature columns. If you have ever used `geom:smooth()`, you have probably used a spline function. The recipes package now has an expanded set of spline functions (with a common naming convention): * [`step_spline_b()`](https://recipes.tidymodels.org/dev/reference/step_spline_b.html) * [`step_spline_convex()`](https://recipes.tidymodels.org/dev/reference/step_spline_convex.html) * [`step_spline_monotone()`](https://recipes.tidymodels.org/dev/reference/step_spline_monotone.html) * [`step_spline_natural()`](https://recipes.tidymodels.org/dev/reference/step_spline_natural.html) * [`step_spline_nonnegative()`](https://recipes.tidymodels.org/dev/reference/step_spline_nonnegative.html) There is also another step to make polynomial functions: [`step_poly_bernstein()`](https://recipes.tidymodels.org/dev/reference/step_poly_bernstein.html) These functions take different approaches to creating the new set of features. Take a look at the references to see the technical details. Here is a simple example using the [Ames data](https://www.tmwr.org/ames.html) where we model the sale price as a nonlinear function of the longitude using a convex basis function: data(ames) ames$Sale_Price <- log10(ames$Sale_Price) spline_rec <- recipe(Sale_Price ~ Longitude, data = ames) %>% step_spline_convex(Longitude, deg_free = 25) spline_fit <- spline_rec %>% workflow( linear_reg() ) %>% fit(data = ames) spline_fit %>% augment(ames) %>% ggplot(aes(Longitude)) + geom_point(aes(y = Sale_Price), alpha = 1 / 3) + geom_line(aes(y = .pred), col = "red", lwd = 1.5) ![](figs/Longitude-1.svg) Not too bad but the model clearly over-fits on the extreme right tail of the predictor distribution. Acknowledgements[](#acknowledgements) -------------------------------------- It’s important that we thank everyone in the community that contributed to tidymodels: * agua: [@coforfe](https://github.com/coforfe) , [@gouthaman87](https://github.com/gouthaman87) , [@jeliason](https://github.com/jeliason) , [@qiushiyan](https://github.com/qiushiyan) , and [@topepo](https://github.com/topepo) . * applicable: [@topepo](https://github.com/topepo) . * bonsai: [@barrettlayman](https://github.com/barrettlayman) , [@DesmondChoy](https://github.com/DesmondChoy) , [@dfsnow](https://github.com/dfsnow) , [@dpprdan](https://github.com/dpprdan) , [@jameslamb](https://github.com/jameslamb) , and [@simonpcouch](https://github.com/simonpcouch) . * broom: [@AaronRendahl](https://github.com/AaronRendahl) , [@AmeliaMN](https://github.com/AmeliaMN) , [@bbolker](https://github.com/bbolker) , [@capnrefsmmat](https://github.com/capnrefsmmat) , [@corybrunson](https://github.com/corybrunson) , [@ddsjoberg](https://github.com/ddsjoberg) , [@friendly](https://github.com/friendly) , [@johanneskoch94](https://github.com/johanneskoch94) , and [@simonpcouch](https://github.com/simonpcouch) . * brulee: [@topepo](https://github.com/topepo) . * butcher: [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@juliasilge](https://github.com/juliasilge) , and [@topepo](https://github.com/topepo) . * censored: [@bcjaeger](https://github.com/bcjaeger) , [@hfrick](https://github.com/hfrick) , [@mattwarkentin](https://github.com/mattwarkentin) , [@simonpcouch](https://github.com/simonpcouch) , [@therneau](https://github.com/therneau) , and [@topepo](https://github.com/topepo) . * corrr: [@jagodap](https://github.com/jagodap) , and [@topepo](https://github.com/topepo) . * finetune: [@DMozzanica](https://github.com/DMozzanica) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@qiushiyan](https://github.com/qiushiyan) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * modeldata: [@topepo](https://github.com/topepo) . * modeldb: [@topepo](https://github.com/topepo) . * parsnip: [@barnabywalker](https://github.com/barnabywalker) , [@bcjaeger](https://github.com/bcjaeger) , [@ben-e](https://github.com/ben-e) , [@CarmenCiardiello](https://github.com/CarmenCiardiello) , [@daniel-althoff](https://github.com/daniel-althoff) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@gustavomodelli](https://github.com/gustavomodelli) , [@hfrick](https://github.com/hfrick) , [@joeycouse](https://github.com/joeycouse) , [@john-b-edwards](https://github.com/john-b-edwards) , [@juliasilge](https://github.com/juliasilge) , [@mdneuzerling](https://github.com/mdneuzerling) , [@mikemahoney218](https://github.com/mikemahoney218) , [@mrkaye97](https://github.com/mrkaye97) , [@qiushiyan](https://github.com/qiushiyan) , [@siegfried](https://github.com/siegfried) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , [@TylerGrantSmith](https://github.com/TylerGrantSmith) , and [@zhaoliang0302](https://github.com/zhaoliang0302) . * plsmod: [@topepo](https://github.com/topepo) . * poissonreg: [@hfrick](https://github.com/hfrick) , [@mattwarkentin](https://github.com/mattwarkentin) , and [@simonpcouch](https://github.com/simonpcouch) . * probably: [@DavisVaughan](https://github.com/DavisVaughan) . * recipes: [@abichat](https://github.com/abichat) , [@adisarid](https://github.com/adisarid) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@JamesHWade](https://github.com/JamesHWade) , [@juliasilge](https://github.com/juliasilge) , [@luisDVA](https://github.com/luisDVA) , [@naveranoc](https://github.com/naveranoc) , [@nhward](https://github.com/nhward) , [@RMHogervorst](https://github.com/RMHogervorst) , [@ruddnr](https://github.com/ruddnr) , [@simonpcouch](https://github.com/simonpcouch) , [@topepo](https://github.com/topepo) , and [@zhaoliang0302](https://github.com/zhaoliang0302) . * rsample: [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hfrick](https://github.com/hfrick) , [@juliasilge](https://github.com/juliasilge) , [@mikemahoney218](https://github.com/mikemahoney218) , and [@tjmahr](https://github.com/tjmahr) . * spatialsample: [@mikemahoney218](https://github.com/mikemahoney218) . * textrecipes: [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@hadley](https://github.com/hadley) , and [@PursuitOfDataScience](https://github.com/PursuitOfDataScience) . * tune: [@Athospd](https://github.com/Athospd) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@frankhezemans](https://github.com/frankhezemans) , [@kevin199011](https://github.com/kevin199011) , [@misken](https://github.com/misken) , [@PursuitOfDataScience](https://github.com/PursuitOfDataScience) , [@qiushiyan](https://github.com/qiushiyan) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * workflows: [@DavisVaughan](https://github.com/DavisVaughan) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@JosiahParry](https://github.com/JosiahParry) , [@lrossouw](https://github.com/lrossouw) , [@msberends](https://github.com/msberends) , [@simonpcouch](https://github.com/simonpcouch) , and [@topepo](https://github.com/topepo) . * yardstick: [@DavisVaughan](https://github.com/DavisVaughan) , [@deschen1](https://github.com/deschen1) , [@EmilHvitfeldt](https://github.com/EmilHvitfeldt) , [@juliasilge](https://github.com/juliasilge) , [@leonhGeis](https://github.com/leonhGeis) , [@PriitPaluoja](https://github.com/PriitPaluoja) , and [@topepo](https://github.com/topepo) . Contents The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # model [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Model [![](/blog/2022/11/model-calibration/thumbnail-sq.jpg)](/blog/2022/11/model-calibration/) [Model Calibration](/blog/2022/11/model-calibration/) [package](/categories/package) Edgar Ruiz Model Calibration is coming to tidymodels. This post covers the new plotting functions, and our plans for future enhancements. [Read more ...](/blog/2022/11/model-calibration/) 2022/11/29 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # h2o [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) H2o [![](/blog/2022/10/tidymodels-2022-q3/thumbnail-sq.jpg)](/blog/2022/10/tidymodels-2022-q3/) [Q3 2022 tidymodels digest](/blog/2022/10/tidymodels-2022-q3/) [roundup](/categories/roundup) Max Kuhn Our post-RStudio conference productivity has been high! This post talks about tidymodels updates from the last few months. [Read more ...](/blog/2022/10/tidymodels-2022-q3/) 2022/10/19 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # lifecycle [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Lifecycle [![](/blog/2022/10/tidyselect-1-2-0/thumbnail-sq.jpg)](/blog/2022/10/tidyselect-1-2-0/) [tidyselect 1.2.0](/blog/2022/10/tidyselect-1-2-0/) [package](/categories/package) Lionel Henry and Hadley Wickham tidyselect 1.2.0 hit CRAN last week and includes a few updates to the syntax of selections in tidyverse functions. [Read more ...](/blog/2022/10/tidyselect-1-2-0/) 2022/10/18 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) --- # clock [Tidyverse](/) [Packages](/packages/) [Blog](/blog/) [Learn](/learn/) [Help](/help/) [Contribute](/contribute/) Clock [![](/blog/2022/09/its-about-time/thumbnail-sq.jpg)](/blog/2022/09/its-about-time/) [It’s about time](/blog/2022/09/its-about-time/) [learn](/categories/learn) Mara Averick Davis Vaughan’s talk from rstudio::conf(2022) on clock, an R package that aims to provide comprehensive and safe handling of date-times. [Read more ...](/blog/2022/09/its-about-time/) 2022/09/28 Categories [deep-dive](/categories/deep-dive) [learn](/categories/learn) [other](/categories/other) [package](/categories/package) [programming](/categories/programming) [roundup](/categories/roundup) The tidyverse is proudly supported by[](https://posit.co/) [Privacy policy](https://www.tidyverse.org/google_privacy_policy) [](https://github.com/tidyverse) [](https://twitter.com/hashtag/rstats) ---