With Great Freedom…

Non-Standard Evaluation is a pretty controversial topic in R circles, and even in the R documentation. Whether you like it never, sometimes, or always, is neither here nor there. What matters is that R allows it. Not many languages give the programmer the power to implement, use, and abuse Non-Standard Evaluation (“NSE”), or anything like it.

So what is NSE? Very roughly, it is to programmatically modify an expression or its meaning after it is issued but before it is executed. You can think of an “expression” an R command you might type at the prompt or in an R script¹. For example in:

subset(mtcars, hp > 250)

                mpg cyl disp  hp drat   wt qsec vs am gear carb
Ford Pantera L 15.8   8  351 264 4.22 3.17 14.5  0  1    5    4
Maserati Bora  15.0   8  301 335 3.54 3.57 14.6  0  1    5    8

subset intercepts the expression hp > 250 before it is run, and changes its meaning by allowing the name hp to resolve against columns from mtcars instead of just against objects in the workspace. In other words, subset performs non-standard evaluation on the expression hp > 250.

Compare to what happens with an expression that is evaluated in the standard way:

mtcars[hp > 250,]

Error in `[.data.frame`(mtcars, hp > 250, ): object 'hp' not found

We get an error because hp is not defined in my workspace.

Standard Evaluation

When we type a simple expression at the R prompt and hit ENTER, R computes its value (a.k.a. evaluates it):

w <- c("am", "I", "global")
rev(w)    # reverse the order of `w`

[1] "global" "I"      "am"    

Ironically the process of evaluation in R is mostly about looking things up rather than computing them. In our example, when we hit ENTER R first looks for the named objects in our expression: rev, and w². Lookups are done through data structures called environments, represented in blue in this flipbook:

After the lookup rev(w) becomes:

(function(x) UseMethod("rev"))(c("am", "I", "global"))

rev is replaced by the definition of the function from the base environment, and w by the character vector from the workspace. The workspace, also known as the global environment, is where name -> value mappings created at the R prompt are kept (e.g. w <- c("am", "I", "global")). Our substituted expression is a bit weird in appearance, but more useful to R as the names rev and w have no inherent meaning.

There is a fair bit of hand waving here³, but for the most part the lookup-and-substitute model reflects observed R behavior.

R is not yet done with our expression. More on this shortly, but first lets talk about environments.

Environments

Environments are akin to named lists with a few additional features. All elements are uniquely named, and the names can be hashed for fast lookups. The name -> object mappings are known as the “Frame” of the environment. Environments also carry a link to an “Enclosing Environment” a.k.a. “Enclosure” (black arrows in the flipbook, pointing to the Enclosure). The Frame and the link to the Enclosure together make up the environment.

WARNING: Environments have reference semantics. Be sure you understand the documentation before you attempt to directly modify elements in environments, e.g. as in env$name[3] <- 42. We do not write to environments here so this isn’t important for our discussion, but be wary when interacting with them directly.

R searches for a name in an environment’s Frame, and if it doesn’t find it moves on to its Enclosure. Enclosures are environments so they too carry a link to their own Enclosure. R repeats the search process until the name is found or it hits the empty environment, which does not have an Enclosure⁴. We’ll call an environment and the sequence of Enclosures it links to an “Environment Chain”⁵.

For expressions typed at the prompt the Environment Chain usually starts at the global environment and links the environments of all the attached packages⁶ (represented with the ... below) through to the base package. When searching for rev, R will work through this entire chain and retrieve the base::rev function from the base package⁷.

When searching for w R will find it immediately in the global environment and stop the search:

The first environment in the Environment Chain is called the “Evaluation Environment”⁸. When we say that expressions are “evaluated in” the Evaluation Environment we mean that they are evaluated according to the Environment Chain that starts with that environment.

When evaluating expressions at the R prompt, the Evaluation Environment is usually the global environment, so this may all seem moot. However, expressions can be issued in different Evaluation Environments, as is the case when they are part of the body of a function.

For a more complete treatment of environments and how they are used by R do read Saruj Gupta’s excellent “How R Searches and Finds Stuff”.

In Functions

After the initial substitution of rev and w for the function and its argument, R will call (i.e “run”) the function. This means repeating the lookup-and-substitute process on any R expressions contained in the body of the function⁹, though with some additional wrinkles.

Most functions in R are “closures”. They are so called because they carry a reference to a “Function Environment”¹⁰ to use as an Enclosure. The Function Environment is typically the Evaluation Environment the function was defined in¹¹. Other functions like the basic arithmetic operators are “primitives”. They are entry points into statically compiled machine code, and contain no R expressions or environments¹².

Each time a closure is called a new environment is created, enclosed by the Function Environment, and with the function parameter -> value¹³ mappings as its Frame. This new environment becomes the Evaluation Environment for the expressions in the function body:

Once the function Evaluation Environment is set up (<rev> in the diagram¹⁴), R can proceed with the lookup-and-substitute process on the body of the function. In this case there is just the one expression UseMethod("rev") in the body.

The key observation is that the Environment Chain changed. It used to start at the global environment and pass through all the loaded packages (<...> in the diagrams). Now it starts at <rev> and goes directly to the base namespace environment as that is where rev is defined. Even though the Environment Chain no longer passes through the global environment, we can still access a copy of the object referenced by w via x in <rev>¹⁵.

On the left is the Environment Chain used to evaluate rev(w). On the right the one used to evaluate the body of the rev function:

<rev> is now the Evaluation Environment. In this context, the global environment becomes the “Calling Environment”, so named because it was the active Evaluation Environment when rev was called via the expression rev(w).

The lookup-and-substitute-and-call-closures process will continue recursively until we hit primitive functions, at which point actual computations happen in machine code¹⁶.

R’s resetting of Environment Chains to be based on the Function Environment instead of the Calling Environment is known as “Lexical Scope”. The term reflects that the Environment Chain is based on how the functions were “written”, instead of how they were called. This scoping mechanism was inherited by R from Scheme, not from S, as can be seen the original R paper.

Masking

It is possible for different environments in a chain to contain the same name. When this happens the object matched is the one from the first environment along the chain that contains that name. Consider what happens if we nest our simple rev(x) expression in a trivial function:

fun <- function() {
  w <- c("am", "I", "fun")
  rev(w)
}

When we call fun(), R creates a new Evaluation Environment enclosed by the global environment as that is where fun was defined. We also assign a new value for w in that Evaluation Environment. The evaluation chain for rev(w) within fun is different from rev(w) at the prompt:

The evaluator finds w in the function’s evaluation environment instead of the now-masked w in the global environment, and so substitutes a different value for it:

rev(w)

[1] "global" "I"      "am"    

fun()

[1] "fun" "I"   "am" 

Non-Standard Evaluation

One of the simplest NSE functions in R is with:

L <- list(w=c("am", "I", "list"))
with(
  L,
  rev(w)     # same commmand
)

[1] "list" "I"    "am"  

What happened? L was somehow made part of the Environment Chain and the names in rev(w) were matched against it. L masked the name lookup as fun’s Evaluation Environment did, except we didn’t need to define a function.

While the concept is straightforward the execution is more complicated. with needs mechanisms for interrupting the evaluation, masking the active Environment Chain in some way with L, and resuming evaluation. R, being the strange language that it is, provides tools to do all this.

Let’s implement a version of with at the prompt to see how this might be done:

expr  <- quote(rev(w))   # capture expression
Lenv <- list2env(L)      # convert list to env
eval(expr, envir=Lenv)   # invoke the evaluator

[1] "list" "I"    "am"  

quote captures a parsed R expression before the evaluator gets to it. Once captured the expression no longer self-evaluates at the prompt:

expr    # we get expression back, not result of evaluating it

rev(w)

It may not seem like much, but it’s a big deal R allows this: unevaluated R expressions can be directly manipulated by R itself. For now all we care about is that the evaluation is on hold, but you can do some interesting things with this as in the RPN in R post.

list2env creates a new environment with the Calling Environment (global env here) as the enclosure. This is natural because environments are akin to named lists.

Lenv

<environment: 0x7fbbd02741c0>

ls.str(Lenv)

w :  chr [1:3] "am" "I" "list"

parent.env(Lenv)   # parent.env == enclosure

<environment: R_GlobalEnv>

eval provides a mechanism to resume evaluation while explicitly specifying an Evaluation Environment:

eval(expr, envir=Lenv)

[1] "list" "I"    "am"  

This is what the Environment Chain looks like right before substituting the w name¹⁷:

By changing the Environment Chain we made the evaluation “Non-Standard”. This is similar to what functions do, but because we are doing it by interrupting evaluation and manually setting the Evaluation Environment it becomes “Non-Standard”.

Now look at what happens if we add another environment to the chain with the rev name in it:

L2 <- list(rev=toupper)
L2env <- list2env(L2, parent=Lenv)
eval(expr, envir=L2env)

[1] "AM"   "I"    "LIST"

We can change the meaning of both functions and values by modifying the Environment Chain. We set the enclosure to L2env to be Lenv with the parent parameter to list2env¹⁸. We could also have directly added a rev mapping to Lenv.

Manipulating the Environment Chain is not the only way to perform NSE. Anything that changes the meaning of an expression after it is issued relative to what would have happened in standard evaluation is NSE.

NSE In Functions

Compare:

with(L, rev(w))

[1] "list" "I"    "am"  

To our hack-at-the-prompt version:

expr  <- quote(rev(w))
Lenv <- list2env(L)
eval(expr, envir=Lenv)

[1] "list" "I"    "am"  

It would be nice to implement with ourselves, but if we try to use quote inside a function to get what someone types in as the argument to that function we are disappointed:

with2 <- function(data, expr) {
  quote(expr)                 # capture expression
  # rest of function will go here later
}
with2(L, rev(w))

expr

What we want to quote is the expression supplied as the argument expr, not the name expr. Thankfully R in its infinite flexibility provides a mechanism for doing this with substitute:

with2 <- function(data, expr) {
  substitute(expr)
  # rest of function will go here later
}
with2(L, rev(w))

rev(w)

When called within a function on a function parameter, substitute acts like quote except it substitutes the unevaluated expression passed as the argument. This allows us to implement with:

with2 <- function(data, expr) {
  expr2 <- substitute(expr)
  denv <- list2env(data, parent=parent.frame())
  eval(expr2, envir=denv)
}
with2(L, rev(w))

[1] "list" "I"    "am"  

We can do a bit better because eval supports adding a list-like element to the Environment Chain out of the box, saving us the list2env step:

with2 <- function(data, expr) {
  expr2 <- substitute(expr)
  eval(expr2, data, enclos=parent.frame())
}
with2(L, rev(w))

[1] "list" "I"    "am"  

list2env specifies Enclosures with parent=, whereas eval does so with enclos=. This is unfortunately one of those areas where R is not as clear as it could be about the names of things, and there are closely related concepts that should be clearly distinguished¹⁹.

So, what’s the parent.frame() business?

Environmental Dichotomy

substitute allows us to implement functions that perform NSE on their arguments. This is a powerful feature, but with it comes a new class of potential errors. Recall that functions create their own Evaluation Environments, but the expressions that we capture and wish to re-evaluate refer to names in the Calling Environment. For things to work correctly we must hand craft an Environment Chain that connects to the Calling Environment at the correct point.

When parent.frame() is called in a function body, it returns the Calling Environment. with2 uses eval to create an Environment Chain that starts with our list L and with the Calling Environment as the Enclosure:

with2 <- function(data, expr) {
  expr2 <- substitute(expr)
  eval(expr2, data, enclos=parent.frame()) #<< Calling Env as Enclosure
}
with2(L, rev(w))

[1] "list" "I"    "am"  

Compare to what happens when we don’t do this:

with2_bad <- function(data, expr) {
  expr2 <- substitute(expr)
  eval(expr2, data)
}
with2_bad(L, rev(w))

[1] "list" "I"    "am"  

No problem right? Except:

expr2 <- c("pathological", "I", "am")
with2_bad(L, rev(expr2))

expr2(rev)

Wow, what the heck is that? Instead of reversing c("pathological", "I", "am") we reversed the expression rev(expr2). Here is what with2_bad effectively did:

expr2 <- quote(rev(expr2))
expr2

rev(expr2)

rev(expr2)

expr2(rev)

It’s mind boggling enough that we can reverse an unevaluated R expression, but imagine you’re the user and everything was perfectly fine until the fateful day you used the expr2 name in your expression. On the “Principle of Least Surprise” scale this outcome is right there with finding a rat in your toilet bowl.

So what happened? We had a name clash with the expr2 symbol that exists both in with2_bad’s Evaluation Environment and in the Calling Environment. This is what the Environment Chain looks like when eval begins the lookup-and-substitute process on rev(expr2) (<w2_b> is with2_bad’s Evaluation Environment):

When evaluating with with2_bad as the enclosure expr2 resolves to the object in with2_bad, instead to the one in the global environment.

As we saw earlier with2 bypasses its Evaluation Environment when calling eval:

So it works as expected:

expr2 <- c("pathological", "I", "am")
with2(L, rev(expr2))

[1] "am"           "I"            "pathological"

There are other ways things go wrong. Suppose we wanted to use with2_bad inside a function that is careful not to use names used by with2_bad:

friendly_fun <- function(L) {
  z <- c('hello', 'friend', '!')
  with2_bad(L, paste(c(z, w), collapse=' '))
}
friendly_fun(L)

[1] "1 5 9 8 am I list"

What happened? Instead of using the z in friendly_fun’s Evaluation Environment, which in this case is with2_bad’s Calling Environment, we used the z in the global environment. Why? Because the with2_bad’s Function Environment, and hence the Enclosure of its Evaluation Environments, is the global environment. So friendly_fun’s Evaluation Environment (<ffun> below) is not part of the Environment Chain:

Again, this is resolved by explicitly setting the Enclosure in eval to the Calling Environment with parent.frame as with2 does:

friendly_fun <- function(L) {
  z <- c('hello', 'friend', '!')
  with2(L, paste(c(z, w), collapse=' '))
}
friendly_fun(L)

[1] "hello friend ! am I list"

The chain is now as we want it to be:

Advanced NSE

Since we can capture user expressions, we can also edit them before we run them. For example, suppose we want to write a function to sum things in the context of a data frame, by group. It could look something like:

mean_call <- function(expr) {
  call <- quote(mean(NULL))
  call[[2L]] <- expr
  call
}
mean_by_grp <- function(data, expr, grp) {
  call_env <- parent.frame()
  grp <- eval(substitute(grp), data, enclos=call_env)
  data <- split(data, grp)
  expr <- mean_call(substitute(expr))
  res <- setNames(numeric(length(data)), names(data))

  for(i in seq_along(data)) {
    res[[i]] <- eval(expr, data[[i]], enclos=call_env)
  }
  res
}

mean_call takes an unevaluated R expression of the type produced by quote or substitute and wraps it in mean:

expr <- quote(Sepal.Length / Sepal.Width)
expr

Sepal.Length/Sepal.Width

mean_call(expr)

mean(Sepal.Length/Sepal.Width)

mean_by_grp splits the data into groups and then evaluates the above expression in the context of those groups:

mean_by_grp(iris, Sepal.Length / Sepal.Width, Species)

    setosa versicolor  virginica 
      1.47       2.16       2.23 

Don’t worry too much about the details, the important point is that we’re calling eval with enclos=call_env. Previously this ensured everything worked fine, and it seems to here, but what if:

mean <- function(...) stop("Boom")
mean_by_grp(iris, Sepal.Length / Sepal.Width, Species)

Error in mean(Sepal.Length/Sepal.Width): Boom

Normally this problem is obviated by the use of packages and associated namespaces. That ensures that when package functions evaluate they resolve their names without interference from user objects. But this doesn’t work here because we explicitly bypass the normal Evaluation Environment and request evaluation in the Calling Environment.

Let’s examine the modified expression mean_call produces:

mean(Sepal.Length/Sepal.Width)

The problem is that we need mean to be resolved according to our function’s Evaluation Environment, but Sepal.Length/Sepal.Width to be resolved according to the Calling Environment. We can only evaluate an expression in a single environment, so we’re stuck.

We could use base::mean instead of mean, but this would still fail if someone redefined :: in the Calling Environment (and yes, they can). Another option is to manually substitute the actual function instead of the name of the function:

mean_call <- function(expr) {
  call <- quote(NULL(NULL))  # call template
  call[[1L]] <- mean         # actual function object
  call[[2L]] <- expr
  call
}

Let’s pretend our functions here are in a hypothetical package pkg with no dependencies. In that case, this is what the Environment Chain looks like as mean_call is injecting mean into the call template (call, shown as NULL(NULL)) in the second line of the function²⁰:

<mcll> is mean_call’s Evaluation Environment, and <pkg> the package namespace. The latter has for enclosure the base namespace, so the version of mean defined in the global environment does not interfere. We get:

mean_call(expr)

(function (x, ...) 
UseMethod("mean"))(Sepal.Length/Sepal.Width)

This looks ugly, but by pre-resolving mean to the correct function we can safely eval the expression in a different Environment Chain:

<mbg> is the Evaluation Environment of mean_by_group, but that Environment Chain is bypassed by eval to resolve Sepal.Length, Sepal.Width, and / in iris enclosed by the Calling Environment.

Conclusions

Geez, that was a lot less fun than I thought it was going to be when I started writing the post. Trying to keep all those closely related but critically different concepts distinct while remaining faithful²¹ to the documentation was exhausting. I imagine reading through this likely had a similar effect on you, so if you managed to get this far I’ll consider it a small victory.

Unfortunately even the standard evaluation model in R is complex, so messing around with it for NSE is even more so. It is particularly challenging because in many cases incorrect usage of NSE works fine, but then proceeds to break in the most bewildering ways under global usage. Further compounding things is the challenging terminology (looking at you parent.env vs parent.frame vs environment).

I do hope that whatever clarity this post might add on the topic is not terminally muddled by its length. If you have any feedback I’d be happy to hear it.

Appendix

Session Info

sessionInfo()

R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3       bookdown_0.18    digest_0.6.25    later_1.0.0     
 [5] mime_0.9         R6_2.4.1         jsonlite_1.6.1   magrittr_1.5    
 [9] evaluate_0.14    blogdown_0.18    stringi_1.4.6    rlang_0.4.5.9000
[13] rstudioapi_0.11  promises_1.1.0   rmarkdown_2.1    tools_3.6.3     
[17] stringr_1.4.0    servr_0.16       httpuv_1.5.2     xfun_0.12       
[21] compiler_3.6.3   htmltools_0.4.0  knitr_1.28      

This is an HTML Canvas flipbook that displays key frames of an animation in a more controllable manner than in a video player. If you are reading this text your browser does not support HTML Canvas.

    FPS:   #: /   ⤾: