extendr v0.9.0
Improving DX
This is (hopefully) our last minor release before extendr v1.0.0, focused almost entirely on improved developer experience.
This update of extendr comes with a lot of new features we are excited to share. While technically a minor release, some of those new features do involve intentional breaking changes. Ordinarily, breaking changes would belong in major versions, but since we haven’t yet reached v1.0.0 we want to be transparent rather than alarming. We also expedited this release to address non-API changes from R Core’s C API stabilization.
Read the full CHANGELOG.
TL;DR
- Move towards semver (semantic versioning)
- Deprecate
rextendr::document() - Remove
Resultfrom the prelude - Use any error type (e.g.
anyhow::Error) - Use
From/TryFrom
If you are not interested in the nitty-gritty of current changes, feel free to skip to the migration guide. It will explain updates you need to make in your packages to address breaking changes.
A note on semantic versioning
The Rust ecosystem encourages the use of semantic versioning (SemVer) to keep expectations managed.
To oversimplify: any time something in your crate’s API changes, gets removed, or otherwise requires changes to user behavior, that should be a major version increment. If a feature is added, that should be a minor version increment. If a bug is fixed, that is a patch version increment.
All too often in the rust ecosystem a minor or patch version introduces breaking changes. We want to practice what we preach and follow semver to the best of our abilities. You can expect the next time that there is a breaking change, there will be a major version bump!
CRAN and non-API entry points
R Core’s C API is a moving target these days, so we have to be vigilant to keep extendr going, addressing changes to the C API in the main branch using backports. We also want your extendr-powered packages to be as stable as possible. Backports ensure that upgrading extendr should not require your R packages to depend on R 4.6+.
Before extendr v0.8.0, we interfaced with the C API using libR-sys, but this was replaced with extendr-ffi, which contains hand-curated bindings to R’s C API. Backports are defined in extendr-ffi at extendr-ffi/src/backports.rs.
If you encounter a CRAN note or warning that we haven’t addressed, please open an issue or mention it on our Discord.
Custom Error types — anyhow integration
Previously, extendr_api::prelude::Result masked the standard Result type — a bad choice on our part that hurt developer experience. It also introduced unnecessary bloat into the prelude, which should really target symbols unique to extendr. Fortunately, some recent developments in extendr allow you to return anyhow::Result or build your own custom error type with thiserror, for example:
use anyhow::Result;
use extendr_api::prelude::*;
#[extendr]
fn foo() -> Result<()> {
bail!()
}Removal of non-standard methods and traits
With this release, we continue our efforts to move away from custom functions towards more idiomatic Rust. In fact, this release deprecates a lot of methods in favor of standard trait implementations. For example, from_hashmap() and into_hashmap() and from_string() and into_string() are replaced with From/TryFrom implementations. The as_str() method is also deprecated, replaced with the standard library’s as_ref().
Changes to the extendr team
Ilia Kosenkov has passed the {rextender} torch to Blake Vernon. Please join us in extending a heartfelt thank you to Ilia for his years of hard work and dedication. Ilia’s contributions to the project cannot be overstated.
We also want to thank Blake for his R Consortium-sponsored work, which is ongoing!
👋🏼 Self-documenting build
Due to the hard work of Mossa and Blake, extendr has a vastly simplified build and documentation process. In fact, we have deprecated rextendr::document(). Users can now call devtools::document() directly to compile their Rust crate and build their R package, with all linked functions seamlessly written to R/extendr-wrappers.R and registered in NAMESPACE if exported. This makes extendr-powered package development more congruent with a devtools workflow.
What allows for this under the hood is effort offloaded to cargo and Rust. Before, extendr and rextendr did a fairly complicated dynamic-loading-dance, but now we rely instead on a small binary called document.rs that links to the crate’s staticlib. The linking allows us to access extendr-macro metadata directly in Rust, hence bypassing dynamic loading and just doing some basic string manipulation before writing to R/extendr-wrappers.R as part of the package build process.
Migration guide
To help you get your package build up-to-date, so you can benefit from these changes, we have introduced rextendr::update_scaffold(). This is similar to running rextendr::use_extendr() on an existing project, but instead of stepping through overwriting existing scaffolding files one by one, it updates the necessary scaffolding automatically. It also prints an informative update message about next steps.
It is worth mentioning, however, that we placed two very large constraints on this update tool. First, it must not edit your package’s manifest Cargo.toml. Second, it should not edit your vendor directory or anything that might affect your dependencies. These constraints have the consequence that we do not fully automate the update process. Some manual editing is required.
The exact steps to update your package are these:
- Update scaffold with
rextendr::update_scaffold(). - Add linked binary to
Cargo.toml. - Update extendr-api version in
Cargo.tomlwithrextendr::use_crate("extendr-api", version = "0.9.0"). - Update Rust source in
rust/srcto handle deprecated extendr methods. - Re-vendor crates with
rextendr::vendor_crates(). - Re-build and document package with
devtools::document().
There are some “gotchas” you might run into while working through the update that we flag below.
Update scaffold
Once you run update_scaffold(), you will see a list of updated files followed by this (somewhat lengthy) update message:
! If your crate or library name differs from the R package name, please
re-run `update_scaffold()` with explicit `lib_name` and `crate_name`.
! You will also need to update `Cargo.toml` to include the following:
[lib]
crate-type = [ "rlib", "staticlib" ]
[[bin]]
name = 'document'
path = 'document.rs'
[dependencies]
extendr-api = "0.9.0"
ℹ You should now call the following in order:
• use_crate("extendr", version = "0.9.0")
• rextendr::vendor_crates()
• devtools::document()
This tells you more or less everything you need to do (same as listed above), though it assumes you are not using any custom extendr methods that have been deprecated in favor of the Rust standard library.
rextendr::use_extendr() set these for you using a Rust-sanitized version of your R package name, but if for some reason you chose to do otherwise, you will need to make sure and specify these explicitly in update_scaffold(). These are required to make sure the scaffolding files actually reference your crate.
Add linked binary to manifest
The really big, super important, thing to note are the changes to Cargo.toml. In order to take advantage of the new self-documenting build process, you will need to manually update your Cargo.toml. First, you need to add "rlib" to your crate type. This signals that your Rust code should be accessible to the binary. Second, you need to add the [[bin]] section referencing the binary document.rs directly. These edits are essential to the new build process.
Basically, Cargo.toml should look like this:
[lib]
crate-type = ["rlib", "staticlib"]
name = <your lib name>
[[bin]]
name = "document"
path = "document.rs"Update extendr-api version
The next important change is to update your version of extendr to the latest release, 0.9.0. You can do this using our utility function
use_crate("extendr-api", version = "0.9.0")In theory, this should only bump the version, leaving any features or other flags alone, but you should check this if you have any customization of this dependency.
src/.cargo/ and src/vendor/. If these are still in src/, you should remove them before updating.
Update Rust source
The breaking changes to extendr-api that we mentioned above will need to be addressed across your crate’s source code. For this, rust-analyzer is really and truly your friend.
Re-vendor crates
So far, so good! The next step is to re-vendor crates to meet the CRAN requirement. Provided you have resolved all dependency conflicts, this should be as simple as running
rextendr::vendor_crates()vendor_pkgs(), but was renamed in this release to more accurately reflect what it does.
Re-build your package
If you’ve made it this far, congratulations! All that is left is to run devtools::document() to re-build and document your package.
What’s next
We are mainly focused on developer experience at the moment, so these are some of our current priorities.
- Better CRAN submission experience. We want a seamless transition between developer-mode and CRAN-compatible builds, notably improvements to Makevars and vendoring.
- More support for R. The two big ones are better error handling, having Rust panics return as R errors, and support for dots, which would allow us to integrate vctrs. Our support wish list also includes ALTREP and R6 (but these require some more thought and careful planning).
- More ergonomic type mapping. Currently, if you have a custom Rust struct or enum you want to return to R, you have to manually implement
IntoRobj/TryFrom<Robj>yourself. We want a more ergonomic trait-based system so that common mappings (e.g. a Rust enum → R character or factor) require less boilerplate. - Introduce parallelism. Rayon is Rust’s data parallelism library, and we are experimenting with different methods for integrating it into extendr.
- Introduce TypedExternal. Right now,
ExternalPtr<T>in extendr wraps a Rust value as an R external pointer, but it’s essentially untyped from R’s perspective — R just sees a raw pointer.TypedExternalwould attach type information so you can safely pass Rust objects around as R objects without losing track of what type they are.
That said, we take feedback seriously, so if you have a feature request for extendr or rextendr, please post a GitHub issue or come discuss it with us on Discord!