stack(                                          # COMBINE - many ways to do this
  lapply(                                       # APPLY
    split(iris$Sepal.Length, iris$Species),     # SPLIT
    mean  # computation to apply
) )

##   values        ind
## 1  5.006     setosa
## 2  5.936 versicolor
## 3  6.588  virginica

Implementations in Base R

Base R provides some functions that facilitate this type of analysis:

tapply(iris$Sepal.Length, iris$Species, mean)

##     setosa versicolor  virginica 
##      5.006      5.936      6.588

aggregate(iris[-5], iris["Species"], mean)

##      Species Sepal.Length Sepal.Width Petal.Length Petal.Width
## 1     setosa        5.006       3.428        1.462       0.246
## 2 versicolor        5.936       2.770        4.260       1.326
## 3  virginica        6.588       2.974        5.552       2.026

I will not get into much detail about what is going on here other than two highlight some important limitations of the built in approaches:

• tapply only summarizes one vector at a time, and grouping by multiple variables produces a multi-dimensional array rather than a data frame as is often desired
• aggregate applies the same function to every column
• Both tapply and aggregate are simplest to use when the user function returns one value per group; both will still work if the function return multiple values, but additional manipulation is often required to get the desired result

Third party packages

iris.dt <- data.table(iris)

Subset

iris %>% filter(Species=="setosa")   # dplyr
iris.dt[Species=="setosa"]           # data.table

Both dplyr and data.table interpret variable names in the context of the data, much like subset.

Modify Column

iris %>% mutate(Petal.Width=Petal.Width / 2.54) # dplyr
iris.dt[, Petal.Width:=Petal.Width / 2.54]      # data.table

One major difference between the two packages is that data.table can modify objects by reference. This is against the general R philosophy of avoiding side effects, but does have the advantage of being faster as it skips a memory re-allocation. For example, in this case we modified the iris.dt object.

Grouping

The philosophical differences between the two packages become more apparent with this task:

iris %>% group_by(Species) %>% summarise(mean(Sepal.Length))  # dplyr
iris.dt[, mean(Sepal.Length), by=Species]                     # data.table

dplyr appears to favor a grammar that conveys the meaning of the task in something resembling natural language, while data.table is looking for compact expressions that achieve the analytical objective.

Now, let’s compute on groups without aggregating, and then filter the results:

# dplyr
iris %>% group_by(Species) %>% 
  mutate(v1=Sepal.Length/mean(Sepal.Length)) %>% 
  select(Species:v1) %>% filter(v1 > 1.15)

# data.table
iris.dt[, Sepal.Length / mean(Sepal.Length), 
  by=Species][V1 > 1.15]

Here you can see that both dplyr and data.table support chaining, but in somewhat different ways. dplyr can keep chaining with %>%, and data.table can chain [.data.table. The main difference is that dplyr chains for every operation, whereas [.data.table only needs to chain if you need to compute on the result of the by operation.

Indirect Variable Specification

Both dplyr and data.table are designed to primarily work by users specifying the variable names they want to compute on. Sometimes it is desirable to set-up computations that will operate without direct knowledge of the variable names. In this example, we attempt to group by a column specified in a variable and compute the mean of all other columns:

col.to.grp.by <- "Species"   # used by both examples

# dplyr
iris %>% regroup(list(as.name(col.to.grp.by)))
  %>% summarise_each(funs(mean))

# data.table
iris.dt[, lapply(.SD, mean), 
  by=eval(col.to.grp.by)]

So this can be done, but it takes a bit of effort and quickly gets complicated if you are trying to do more interesting computations. Explaining what’s going on here is a topic for another post. Note that the dplyr example doesn’t work with the current version (see discussion on SO).

Parting Thoughts on Syntax / Grammar

As noted earlier data.table favors succint syntax whereas dplyr favors a grammar that more closely follows common language constructs. Which approach is better is ultimately a matter of personal preference. Interestingly both dplyr and data.table depart from the base R paradigms in their own ways. data.table’s syntax is much closer to base R functions, but it gleefully employs side effects to achieve it’s efficiency objectives. dplyr’s grammar is completely different to base R, but it does adhere to the no side effects philosophy.

Dimensions

We will test how the following factors affect performance:

Dimension	Values Tested
Number of rows in data.frame	10K, 100K, 1MM, 10MM (also tried 500K, 5MM, and 20MM)
Number of columns in data.frame	1 or 3 numeric data columns + grouping column
Groups	10, 100, 1000, …, 1MM with min group size == 10 rows
Group size constancy	tested with group sizes exactly as above as well as group sizes on average as above but for any given group random normally distributed with SD == 0.2 mean group size
Operation Type	Aggregating (using mean), Non-aggregating (using rev)

Data

The data set was structured to contain on single factor grouping column under the assumption that grouping by multiple columns is unlikely to be a big driver of performance. The data columns are all numeric. I did not test other data types mostly because numeric columns are the common use case and I was running into waay too many permutations already. Here is an example data frame:

DF

'data.frame': 10000 obs. of  4 variables:
 $ G.10: Factor w/ 1000 levels "1","2","3","4",..: 690 188 414 595 665 933 405 851 516 439 ...
 $ V.1 : num  0.736 0.702 0.691 0.377 0.161 ...
 $ V.2 : num  0.0112 0.0763 0.175 0.3586 0.2254 ...
 $ V.3 : num  0.516 0.268 0.484 0.822 0.989 ...

The G.10 column is the one we will group by. The V.# columns are random uniform in c(0, 1). Data frames will change in size/dimensions as described in the “Dimensions” section above, but will otherwise look pretty much like this one. Here we show that in this example the groups are on average of size 10 rows, but do vary in size.

It turns out that having every group the same size or having them varying in size as shown above has very little impact on performance, so I’m only showing results for the unequal size tests.

Procedure

I pre-generated all the combinations of data as described above, and then ran five iterations of each test for both dplyr and data.table, discarding the slowest of the five tests. Tests were timed with system.time and the values I report are the “elapsed” times.

The computations chosen (mean and rev) are purposefully simple low overhead functions to ensure that the bulk of the execution is related to the splitting and grouping of the data sets and not evaluating the user functions. These are the command I used (one column versions):

# Aggregating

df.sub %>% group_by(X) %>% summarise(mean(V.1))
data.table(df.sub)[, mean(V.1), by=X]

# Non-Aggregating

df.sub %%>%% group_by(X) %>% mutate(res=rev(V.1))
data.table(df.sub)[, res:=rev(V.1), by=X]

The bulk of the tests were run on Mac OS 10.8 with R 3.0.2, dplyr 0.1.3, and data.table 1.9.2 (the vs. base tests were run on a different machine).

You can look at the full data generation and benchmarking code if you wish.

Results

The short of it is that dplyr and data.table perform comparably for all data frame sizes unless you have a very large number of groups. If you do have high group numbers then data.table can be substantially faster, particularly when using computations that lead to one row per group in the result. With fewer groups dplyr is slightly faster.

The following chart summarizes the results on data frames with one column of data. Please note that the X axis in the plots is log10. The facet rows each correspond to a particular data frame size. The columns distinguish between an aggregating operation (left, using mean) and non-aggregating one (right, using rev). Higher values mean slower performance. Each point represents one test run, and the lines connect the means of the four preserved test runs.

dplyr and data.table are neck and neck until about 10K groups. Once you get to 100K groups data.table seems to have a 4-5x speed advantage for grouping operations and 1.5x-2x advantage for non-grouping ones. Interestingly it seems that the number of groups is more meaningful in the performance difference as opposed to the size of the groups.

Adding columns seems to have very little effect on the grouping computation, but a substantial impact on the non-grouping one. Here are the results for the 10MM row data frame with one vs. three columns:

Vs. Base

I also ran some tests for base, focusing only on the 1MM row data frame with 100K groups (note these tests were run on a different machine, though same package versions). Here are the commands I compared:

# Aggregating

aggregate(DF$V.1, DF[1], mean)
data.table(DF)[, mean(V.1), by=G.10]
DF %>% group_by(G.10) %>% summarise(mean(V.1))

# Non-Aggregating

DF$res <- ave(DF$V.1, DF$G.10, FUN=rev)))
data.table(DF)[, res:=rev(V.1), by=G.10]
DF %>% group_by(G.10) %>% mutate(res=rev(V.1))</pre>
And the results:

Suprisingly for non-aggregating tasks base performs remarkably well.