That @anthonynorth has got us covered #rstats
Today I participated in the first meeting of the #rstats RConsortium working group for R repositories. The path I started on with cranchange lead me to this point, although this group has a much larger scope.
On the CRAN side of things I was encouraged to hear from Michael Lawrence that there is a desire to make change at CRAN including plans create a more informative public web presence, and bring on someone in a Developer Advocate role(!).
One thing I think that is going to be key to positive change is eliciting some clearer sense from CRAN as to what the group’s goals and priories are. For example: What priority is placed on being a Continuous Integration service for R-Core vs a validation and distribution mechanism for a rolling release of R packages?
I have a hunch that some of the inconsistency R users and developers see is due to tension between these types of objectives, but I am keen to learn more from this group.
I am very thankful to the Linux Foundation and RConsortium for facilitating this group, especially Joseph Rickert for leading.
Hadley Wickham’s meeting minutes are accessible from the repository
#rstats VSCode productivity tip: assign keybindings to workbench.action.terminal.scrollDown and workbench.action.terminal.scrollUp so you can move though console output without having to switch back and forth from the terminal or use your mouse.
When you take advantage of {dplyr} groups for mutating or filtering you DON’T get the helpful warning about sticky groups as per summarise(). This red flag I put into {paint} has saved me twice in 2 days! #rstats
New in {rmdocs} 0.2.0:
You get an Rmd verison of the dev help for your package when it detects {devtools} is loaded. #rstats
Something in prod bombed over the weekend. I had logged the #rstats {targets} build output so I knew which target. I pulled the dependencies from the cache stepped through the target code interactively until I found the bomb. Data source schema change ofcourse. Took about ~10 minutes to pin down the field and record in the cached data and fire off the email to upsream team. Prod without a target graph and a cache? I can’t even.
Setup
In response to a Twitter question from Jared Lander, here is my logging setup:
Top level file is a .cmd - yes we’re on Windows Server Data Centre.
pushd $~dp0
Rscript.exe the_script.R > log.txt 2>&1
Roughly translated to:
- set the working dir to the scipt’s location.
- pipe the std output and std error of running the_script.R to log.txt
Here is the_script.R:
capsule::run({
targets::tar_invalidate(source_file)
targets::tar_make(output)
})
Which translates to:
- within my {capsule} ({renv}):
- invalidate the source data (so it will be refreshed)
- build the output (this plan has multiple outputs on different schedules)
To do interactive diagnostics with cached targets, I run capusle::repl() to switch my R REPL over to the capsule environment.
After 7 days of dogfooding, the addition of a test suite, documentation, and 1 confirmed user, {paint} is now v0.1.0 #rstats
Happy Friday #rstats! This week’s diversion is alternative print() methods for data rectangles. Here’s {paint}, highly experimental, works on my machine stage:
Bit of a scraping project on that’s going to sip thousands of xml files over the next couple of days. Using #rstats {targets} dynamic branching each xml file is a cached target. The advantage of this is that if the process is interrupted at any time for any reason it can be resumed with a simple tar_make(). Niiiiiiice.
Happy Friday #rstats! Incited by @mdsumner {rmdocs} now provides replacements for utils::help and utils::?, so you never have to accidently look at HTML help again.
New {targets} addin: tflow::rs_load_current_editor_targets(). Load targets from cache that are referred to in the code you are looking at. #rstats
Happy Friday {rmdocs} now handles namespaced::symbols #rstats
If you run the addin with the cursor on withr::with_options you’ll get served some delicious Rmd help for that function, even if you had rlang::with_options masking it via library(rlang).
Unlocking fast #rstats lockfile generation
This week I cracked a problem that I’d been stewing on for a while: Fast generation of renv.lock files.
For those not in the know: These fully describe an R project’s package dependencies and can be used to create a “known good” package environment for the project to run in. You should definitely be using these! Typically these are created with {renv}.
I set myself a budget of 3 seconds to:
- Detect my project dependencies
- Read package metadata
- Determine a full set of recursive dependencies
- Write a lock file readable by
{renv}
My thinking was that this amount of time is short enough to facilitate new workflows involving always-on automated lockfile generation. So instead of lockfile creation being a kind of manual discipline that is done interactively, it can become something that just automatically happens everytime you build a pipeline with {targets} or render a document with {rmarkdown}.
And that means people won’t forget to do it before they go on holiday, Murhpy’s law etc etc.
The generation time has to be really short because during the iteration cycle of an analysis you’re typically building a pipeline many many times in a single day. You may be adding or removing dependencies each time. Time spent waiting for things to build can rapidly become annoying, and that annoyance inspires hacks that undermine everything.
Anyway I’m happy to report success. capsule::capshot() can tick all the items I listed off in 1.5 - 2 seconds on my current project which is quite mature and laden with dependencies (~ 200 recursive deps). You give it paths to files containing your dependencies (typically a single file for me), and you get back a lockfile, built against the current .libPaths().
So you’ve likely never heard of {capsule} (although it does have its fans). It’s a kind of reimagining of the {renv} workflow for my team. It actually uses {renv} under the hood. The main point of difference is that it’s a lazy workflow. You don’t typically work out of a local library. You do that only when picking something up that’s been on the shelf for a long time, or putting something “into production” - i.e. running unsupervised somewhere.
The laziness has several advantages: You get no interaction with personal dev setups. RStudio, VSCode, Emacs, Addin packages etc… none of that needs to go anywhere near the lockfile. It’s also an easy sell. There’s 2 commands you absolutely need to know and they have obvious names: capsule::create() and capsule::run().
Some cool opportunities get opened up by the always-make-a-lockfile workflow. If we’re doing that, hopefully, we’re always committing it, and so it can become a mechanism to help nudge team-mates to keep their R libraries moving forward in step.
For example, your lockfile target could pass on building a new lockfile that would contain versions behind the current one, and send a warning to update pacakges. There’s actually machinery already in {capsule} for that, although I am still settling on the best design. I am excited to get a feel for the best practice for this kind of stuff over the next few weeks!
Are CRAN’s policies degrading #rstats package quality?
Due to one of my current projects, R developers have been sharing their frustrations with CRAN with me. There are many distrubing aspects to these stories, but one that is on loop in my brain at the moment is the systemic degradation CRAN policies are creating.
I think this degradation is slow and doesn’t impact too much on functionality, so it will be hard to spot at first. If there is a trend though, over time its corrosive nature will become sorely apparent. This is because developers have confessed to:
- removing all external links from documentation to avoid being flagged when one of those becomes a redirect.
- deleting examples from their code that were being run even though they were flagged with
\donttest. - suppressing tests on CRAN that were creating issues that could not be easily reproduced
- ditching vignettes that were struggling to build on CRAN
It’s kind of sad to imagine what the cumulative effect looks like of developers being nudged away from creating thoroughly tested works with rich interconnected explanatory documentation. To me, it’s just an odd situation to be in: where R itself contains excellent tools for this, but our package infrastructure is having a potentially out-sized influence on the utilisation of those tools.
l33t Data scientists can write #julialang to write R. They can write #haskell to write R. They can write #clojure to write R. For a long time many have been writing a half-baked version of #rstats ported to #python. Life’s a bit simpler if you Just. Write. R. Though?
Just say no to !important
Subtweeting an out of control hairy yak that has raised 2 gh issues, 1 stack overflow question, and 1 package update, and eaten most of my day.
Any old folder can be a git remote
Becasuse of GitHub I am not that used to thinking of git as a peer to peer decentralised version control system - despite the fact I know this theoretically. An upshot of this property that any folder that you have access to can act as a remote.
This came in handy today, rigging up a way to deploy code to a server that has very limited connectivity to the outside world.
First I copied my local copy over to the server.
Then on my workstation I added the folder as a remote:
git remote add prod \\analytics\blah\project_name
Then I can fetch that remote to get the branch metadata:
git fetch prod
after which I can push to it
git push prod
Well sort of. Initally I got this error:
git push prod
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (11/11), 1.37 KiB | 467.00 KiB/s, done.
Total 11 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Checking connectivity: 11, done.
remote: error: refusing to update checked out branch: refs/heads/main
remote: error: By default, updating the current branch in a non-bare repository
remote: is denied, because it will make the index and work tree inconsistent
remote: with what you pushed, and will require 'git reset --hard' to match
remote: the work tree to HEAD.
remote:
remote: You can set the 'receive.denyCurrentBranch' configuration variable
remote: to 'ignore' or 'warn' in the remote repository to allow pushing into
remote: its current branch; however, this is not recommended unless you
remote: arranged to update its work tree to match what you pushed in some
remote: other way.
remote:
remote: To squelch this message and still keep the default behaviour, set
remote: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
To \\analytics\blah\project_name
! [remote rejected] main -> main (branch is currently checked out)
error: failed to push some refs to '\\analytics\blah\project_name
Git refused to accept my commits. The issue it is warning me about is that if I did push commits onto the main branch, the HEAD pointer in the copy on the server is not updated. So it would need to be updated by running git reset in the server’s copy.
This can be fixed by running this command on my local machine:
git config --global receive.denyCurrentBranch "updateInstead"
So now when I try to push commits onto the current branch in the server’s copy it will automatically update the HEAD, provided there is no other uncommited changes hanging out in the working tree. RAD.
So when @github Copilot drops a line of code into my project it’s also going to drop in 200 million LICENSE files and attributions right? RIGHT?
We have access to Pfizer vaccine at my work. I reckon you could draw a line through the department separating who’s vaxxed and who’s ‘gonna do it soon’ by asking ‘Can you describe an expontential function?’.
My review of the ZSA Moonlander Ergonomic Keyboard: www.milesmcbain.com/posts/zsa…
Another riff on the placeholder idea with |>
Another riff on the placeholder idea with |>:
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
. <- function(.dat, template){
template_code <- deparse(substitute(template))
arg <- deparse(substitute(.dat))
interpolated_code <- gsub("(?<=[(, ])?[.](?=[), \\[])", arg, template_code, perl = TRUE)
eval(parse( text = interpolated_code))
}
"a" |>
.(c(., "b")) |>
.(setNames(., .))
#> a b
#> "a" "b"
mtcars |>
transform(kmL = mpg / 2.35) |>
.(lm(kmL ~ hp, data = .))
#>
#> Call:
#> lm(formula = kmL ~ hp, data = transform(mtcars, kmL = mpg/2.35))
#>
#> Coefficients:
#> (Intercept) hp
#> 12.80803 -0.02903
"col_name" |>
.(mutate(mtcars, . = "cool")) |>
.(bind_cols(., .)) |>
.(.[1, ])
#> New names:
#> * mpg -> mpg...1
#> * cyl -> cyl...2
#> * disp -> disp...3
#> * hp -> hp...4
#> * drat -> drat...5
#> * ...
#> mpg...1 cyl...2 disp...3 hp...4 drat...5 wt...6 qsec...7 vs...8
#> Mazda RX4 21 6 160 110 3.9 2.62 16.46 0
#> am...9 gear...10 carb...11 col_name...12 mpg...13 cyl...14 disp...15
#> Mazda RX4 1 4 4 cool 21 6 160
#> hp...16 drat...17 wt...18 qsec...19 vs...20 am...21 gear...22
#> Mazda RX4 110 3.9 2.62 16.46 0 1 4
#> carb...23 col_name...24
#> Mazda RX4 4 cool
Created on 2021-06-24 by the reprex package (v2.0.0)
I call . the ‘neutering’ function.
How you’d fix the #rstats dog’s balls pattern
The dog’s balls pattern is a thing. I didn’t name it.
This is the pattern:
mtcars |>
transform(kmL = mpg / 2.35) |>
( \(df)
lm(kmL ~ hp, data = df)
)()
Copy pasta from this tweet.
Noisy syntax involving parentheses, including a werid empty pair hanging out in the breeze at the end. The easiest thing for beginners anyone to forget or accidentally unbalance.
So rather than reinvent the wheel, let’s take a quick look at how other programming languages with pipes have solved this issue.
Well there’s the Hack pipe and it uses a $$ placeholder to allow the user to set the position without making a lambda:
$x = vec[2,1,3]
|> Vec\map($$, $a ==> $a * $a)
|> Vec\sort($$);
But Hack? That’s a bit obscure.
What about Julia? Something more data sciencey and close to home. Well Julia uses a @pipe macro to, you guessed it, let the user deploy a placeholder to the arg position to be piped to:
@pipe a |> addX(_,6) + divY(4,_) |> println # 10.0
This macro theme is repeated in other languages. Checkout Clojure, it has so many pipes: -> pipe to first, ->> pipe to last, and ofcourse, as-> pipe to placeholder.
Okay so I am just cherry-picking examples. But the placeholder or placeholder/macro combination is a solution with precedent to the problem of how to pipe into an argument other than the first.
So let’s think now about R. We don’t have macros. Game over? No. R’s famed syntax malleability via lazy evaluation and syntax tree operations is how we get that kind of stuff done.
To fix Dog’s balls we’d be looking at some kind of function that manipulates the syntax tree. That is to say, it can turn:
a |> b(x, _) into a |> b(x, a)
Clearly, it needs to know about the symbols a and b(x, _) so it has to be an infix operator. Something like:
a %|>% b(x, _)
Where the %|>% function’s job is to rewrite the syntax tree by replacing any _ in the tree on its right-hand side, with the thing on its left-hand side. Easy done? Well, there is a recursion issue. It needs to rewrite:
a %|>% b(x, _) %|>% c(y, _) into c(y, b(x, a)) but details details.
I do think we can probably shave down some characters…. maybe drop the |? Still keeps the forward idea going.
And how do we feel about _… a bit Pearl-ish… maybe ? hmmm no that doesn’t inspire confidence… . ahhhh brief but firm - I like it. Putting it all together we have our new pipe:
a %>% b(x, .)
Now, I already know what you’re going to say, “This is not a pipe”.