A fuzzy monster with a lasso and a cowboy hat using it to wrangle other fuzzy monsters, with the styled text dplyr: go wrangling

Figure from Allison Horst

Wrangling

Introduction

This is a new lecture from the previous delivery of this course. In the last offering, I found that the process of wrangling data was by far the thing that people had the most trouble with. In recitations, and for module assignments, I would provide data in a way that would need some adjustment before visualization can be made - and if I’m being honest, I heard a lot of rumblings about this.

Still, I am going to leave in the course activities that required data to the wrangled before visualization. I am doing this because real data is mostly not structured precisely how it needs to be to make the visualizations you want. I want to provide you all some practice to get comfortable with using your data lassos. This is something you need to get comfortable with on your coding journey.

But, I have added in this extra lecture to explicitly go over what I think are the most useful wrangling functions and tools you can use in R. I hope this introduces you to some of what is possible with R, so it will trigger your memory later when you need to use it. You can also always come back to this page during the course.

What is the tidyverse?

“The tidyverse” is a collection of packages called that are designed for data science. You can certainly use R without using the tidyverse, but it has many packages that I think will make your life a lot easier. We will be using mostly tidyverse functions in this class, with some base R syntax scattered throughout.

The “core tidyverse” contains the 8 packages below:

  • dplyr: for data manipulation
  • ggplot2: a “grammar of graphics” for creating beautiful plots
  • readr: for reading in rectangular data (i.e., Excel-style formatting)
  • tibble: using tibbles as modern/better dataframes
  • stringr: handling strings (i.e., text or stuff in quotes)
  • forcats: for handling categorical variables (i.e., factors) (meow!)
  • tidyr: to make “tidy data”
  • purrr: for enhancing functional programming (also meow!)

We will be using many of these other packages in this course, but will talk about them as we go. There are more tidyverse packages outside of these core eight, and we will talk about some of them another time.

tl;dr Tidyverse has a lot of packages that make data analysis easier. None of them are required, but I think you’ll find many tidyverse approaches easier and more intuitive than using base R.

You can find here some examples of comparing tidyverse and base R syntax.

Today we will be mostly talking through functions that live within the dplyr package.

Installing ggplot & tidyverse

To install packages in R that are on the Comprehensive R Archive Network (CRAN), you can use the function install.packages().

install.packages("tidyverse")
install.packages("ggplot2")

We only need to install packages once. But, every time we want to use them, we need to “load” them, and can do this using the function library().

library(tidyverse)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.2     ✔ readr     2.1.4
## ✔ forcats   1.0.0     ✔ stringr   1.5.0
## ✔ ggplot2   3.4.2     ✔ tibble    3.2.1
## ✔ lubridate 1.9.2     ✔ tidyr     1.3.0
## ✔ purrr     1.0.1     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

It’s a good habit to not ignore warnings/messages that R gives you.

tl:dr install.packages() once, library() every time.

Loading data

In class, we will use a combination of data embedded within R (or packages in R), from the internet, or data you import yourself. I am going to quickly go over ways to import common data types.

.csv

Files saved as comma separated values are the most common data type I tend to import. The function read_csv() which is a part of the tidyverse package readr allows you to do this easily as it has a special function for this file type, as it is so common.

Make sure that your file is within your working directory (or you have its relative or complete path), and you can install it (and save it) like this:

sample_csv_data <- read_csv(file = "my-file-name.csv")

.xlsx

The second most common file type I import are those made in Excel. These files can either be converted to a .csv and then read in like we just went over, or you can load the package readxl and read files in directly. If you don’t already have readxl you can download it using install.packages().

library(readxl)
sample_excel_data <- read_excel(file = "my-file-name.xlsx",
                                sheet = "Sheet1")

Here you can find the readr cheatsheet.

The pipe |>

The pipe |> (which used to be written %>%, and you will see this widely when googling/troubleshooting and sometimes see me default to this older syntax) is a tool that allows you to take the output of one function, and send it to the next function.

You can read the pipe as “and then” - here is a theoretical example.

take_this_data |>
  then_this_function() |>
  then_another_function() |> 
  finally_a_last_function()

The easiest way to see how the pipe works is with an example. We are going to use the dataset diamonds which comes pre-loaded when you load the tidyverse.

What is in the dataset diamonds? We can get a “glimpse” of it with the function glimpse, which is sort of like the tidyverse version of str().

glimpse(diamonds)
## Rows: 53,940
## Columns: 10
## $ carat   <dbl> 0.23, 0.21, 0.23, 0.29, 0.31, 0.24, 0.24, 0.26, 0.22, 0.23, 0.…
## $ cut     <ord> Ideal, Premium, Good, Premium, Good, Very Good, Very Good, Ver…
## $ color   <ord> E, E, E, I, J, J, I, H, E, H, J, J, F, J, E, E, I, J, J, J, I,…
## $ clarity <ord> SI2, SI1, VS1, VS2, SI2, VVS2, VVS1, SI1, VS2, VS1, SI1, VS1, …
## $ depth   <dbl> 61.5, 59.8, 56.9, 62.4, 63.3, 62.8, 62.3, 61.9, 65.1, 59.4, 64…
## $ table   <dbl> 55, 61, 65, 58, 58, 57, 57, 55, 61, 61, 55, 56, 61, 54, 62, 58…
## $ price   <int> 326, 326, 327, 334, 335, 336, 336, 337, 337, 338, 339, 340, 34…
## $ x       <dbl> 3.95, 3.89, 4.05, 4.20, 4.34, 3.94, 3.95, 4.07, 3.87, 4.00, 4.…
## $ y       <dbl> 3.98, 3.84, 4.07, 4.23, 4.35, 3.96, 3.98, 4.11, 3.78, 4.05, 4.…
## $ z       <dbl> 2.43, 2.31, 2.31, 2.63, 2.75, 2.48, 2.47, 2.53, 2.49, 2.39, 2.…

What if we want to see what is the average price of a diamond where cut = "Premium". There are a few ways we can do this.

# one way
# filter for only the premium diamonds
diamonds_premium <- filter(diamonds, cut == "Premium")

# calculate the mean using summarize
summarize(diamonds_premium, mean_price = mean(price))
## # A tibble: 1 × 1
##   mean_price
##        <dbl>
## 1      4584.
# or calculate mean using mean
# the function mean() requires a vector
mean(diamonds_premium$price)
## [1] 4584.258

Or, we can use the pipe |>. We are going to talk about summarize() in a minute.

diamonds |>
  filter(cut == "Premium") |>
  summarize(mean_price = mean(price))
## # A tibble: 1 × 1
##   mean_price
##        <dbl>
## 1      4584.
# if we want to use the function mean() we need to supply a vector
diamonds |> 
  filter(cut == "Premium") |>
  pull(price) |> # pulls out price as a vector
  mean()
## [1] 4584.258

Some reasons I like the pipe:

  • its easier to read (and doesn’t have a lot of nested parentheses)
  • it doesn’t require you to create lots of interim objects which you won’t use again
  • its easy to troubleshoot

The keyboard shortcut for |> is Ctrl/Cmd + Shift + M

Of course you can assign the output of a pipe to something using the assignment operator <- and then use it for other things.

Ssome functions are not “pipe friendly” meaning they will not work using pipes. This is often because the data is not the first argument passed to the function. All tidyverse functions work with piping.

Selecting columns with select()

Often you will want to pick only certain columns in your dataframe, and you can do this with the function select(). You can pick columns by:

  • their names
  • their position (i.e., index)
  • characteristics of that column

Let’s select first by name.

diamonds |> 
  select(carat, cut, price)
## # A tibble: 53,940 × 3
##    carat cut       price
##    <dbl> <ord>     <int>
##  1  0.23 Ideal       326
##  2  0.21 Premium     326
##  3  0.23 Good        327
##  4  0.29 Premium     334
##  5  0.31 Good        335
##  6  0.24 Very Good   336
##  7  0.24 Very Good   336
##  8  0.26 Very Good   337
##  9  0.22 Fair        337
## 10  0.23 Very Good   338
## # ℹ 53,930 more rows

Note that when you use the pipe, the potential column names will autofill for you after you type 3 letters. You can also hit tab to scroll through all the potential objects to select.

We can also select by index. In general I would recommend against this because its really hard to remember which column indices are which variables today, nevermind returning back to old code 1 year from now.

diamonds |> 
  select(c(1, 2, 7)) # you could also use the colon syntax if your columns are sequential
## # A tibble: 53,940 × 3
##    carat cut       price
##    <dbl> <ord>     <int>
##  1  0.23 Ideal       326
##  2  0.21 Premium     326
##  3  0.23 Good        327
##  4  0.29 Premium     334
##  5  0.31 Good        335
##  6  0.24 Very Good   336
##  7  0.24 Very Good   336
##  8  0.26 Very Good   337
##  9  0.22 Fair        337
## 10  0.23 Very Good   338
## # ℹ 53,930 more rows

You can also select using selection helpers like:

Here is an example of using where() to select only the columns that are numeric.

diamonds |> 
  select(where(is.numeric))
## # A tibble: 53,940 × 7
##    carat depth table price     x     y     z
##    <dbl> <dbl> <dbl> <int> <dbl> <dbl> <dbl>
##  1  0.23  61.5    55   326  3.95  3.98  2.43
##  2  0.21  59.8    61   326  3.89  3.84  2.31
##  3  0.23  56.9    65   327  4.05  4.07  2.31
##  4  0.29  62.4    58   334  4.2   4.23  2.63
##  5  0.31  63.3    58   335  4.34  4.35  2.75
##  6  0.24  62.8    57   336  3.94  3.96  2.48
##  7  0.24  62.3    57   336  3.95  3.98  2.47
##  8  0.26  61.9    55   337  4.07  4.11  2.53
##  9  0.22  65.1    61   337  3.87  3.78  2.49
## 10  0.23  59.4    61   338  4     4.05  2.39
## # ℹ 53,930 more rows

You can find more helpers here.

Using select() will also set the order of your columns. More about this later.

Choosing observations with filter()

Cartoon showing three fuzzy monsters either selecting or crossing out rows of a data table. If the type of animal in the table is “otter” and the site is “bay”, a monster is drawing a purple rectangle around the row. If those conditions are not met, another monster is putting a line through the column indicating it will be excluded. Stylized text reads “dplyr::filter() - keep rows that satisfy your conditions.” Learn more about dplyr::filter.

Figure from Allison Horst

Sometimes you want to select observations (rows) based on values. To do this you use filter(). Try not to confuse this with select().

select() picks columns, while filter() picks rows.

The function filter() will keep only observations that meet your filtering criteria.

Let’s say we want to only keep the diamonds that are bigger than 3 carats.

diamonds |> 
  filter(carat > 3)
## # A tibble: 32 × 10
##    carat cut     color clarity depth table price     x     y     z
##    <dbl> <ord>   <ord> <ord>   <dbl> <dbl> <int> <dbl> <dbl> <dbl>
##  1  3.01 Premium I     I1       62.7    58  8040  9.1   8.97  5.67
##  2  3.11 Fair    J     I1       65.9    57  9823  9.15  9.02  5.98
##  3  3.01 Premium F     I1       62.2    56  9925  9.24  9.13  5.73
##  4  3.05 Premium E     I1       60.9    58 10453  9.26  9.25  5.66
##  5  3.02 Fair    I     I1       65.2    56 10577  9.11  9.02  5.91
##  6  3.01 Fair    H     I1       56.1    62 10761  9.54  9.38  5.31
##  7  3.65 Fair    H     I1       67.1    53 11668  9.53  9.48  6.38
##  8  3.24 Premium H     I1       62.1    58 12300  9.44  9.4   5.85
##  9  3.22 Ideal   I     I1       62.6    55 12545  9.49  9.42  5.92
## 10  3.5  Ideal   H     I1       62.8    57 12587  9.65  9.59  6.03
## # ℹ 22 more rows

Here I made use of the greater than > sign, and there are other operators you could also use to help you filter.

  • ==: equal to (I usually read this as exactly equal to, and is different than using an equal sign in an equation)
  • <, >: less than or greater than
  • <=, >=: less than or equal to, great than or equal to
  • &: and
  • |: or
  • !: not equal
  • is.na: is NA

You can also layer your filtering.

diamonds |> 
  filter(carat > 3 & cut == "Premium")
## # A tibble: 13 × 10
##    carat cut     color clarity depth table price     x     y     z
##    <dbl> <ord>   <ord> <ord>   <dbl> <dbl> <int> <dbl> <dbl> <dbl>
##  1  3.01 Premium I     I1       62.7    58  8040  9.1   8.97  5.67
##  2  3.01 Premium F     I1       62.2    56  9925  9.24  9.13  5.73
##  3  3.05 Premium E     I1       60.9    58 10453  9.26  9.25  5.66
##  4  3.24 Premium H     I1       62.1    58 12300  9.44  9.4   5.85
##  5  3.01 Premium G     SI2      59.8    58 14220  9.44  9.37  5.62
##  6  4.01 Premium I     I1       61      61 15223 10.1  10.1   6.17
##  7  4.01 Premium J     I1       62.5    62 15223 10.0   9.94  6.24
##  8  3.67 Premium I     I1       62.4    56 16193  9.86  9.81  6.13
##  9  3.01 Premium I     SI2      60.2    59 18242  9.36  9.31  5.62
## 10  3.04 Premium I     SI2      59.3    60 18559  9.51  9.46  5.62
## 11  3.51 Premium J     VS2      62.5    59 18701  9.66  9.63  6.03
## 12  3.01 Premium J     SI2      60.7    59 18710  9.35  9.22  5.64
## 13  3.01 Premium J     SI2      59.7    58 18710  9.41  9.32  5.59

Make new columns with mutate()