Albert Rapp

WebDev4R: Shiny Explained

Albert Rapp — Sat, 13 Sep 2025 22:00:00 GMT

Shiny in a Nutshell

Video

Checkout the full R-Shiny playlist here 👉 PLAYLIST

Code

Press to unfold code

How to Finally Fix Your Tables in Typst PDFs (using {gt} tables)

Albert Rapp — Sat, 30 Aug 2025 22:00:00 GMT

The Easiest Way to Export Data From Your R-Shiny Apps

Video

Checkout the my other explainer videos here 👉 PLAYLIST

Code

All code in additional files.

Additional Files

File Management With The {fs} Package

Albert Rapp — Sat, 29 Mar 2025 23:00:00 GMT

As data scientists we often have to deal with lots of tedious tasks. One such tedious task can be interacting with the file system on our computer or the remote machine we’re working with. Thankfully, the {fs} package has a bunch of convenvience function that make our life a whole lot easier.

Let’s check out a few examples. And if videos are more your thing, you can also watch the video version of this blog post on YouTube.

Assemble paths

Check out this data set.

library(tidyverse)
library(fs)
original_tib <- tibble(
  dir = c('some/path/blub', 'bla/here/', 'direct/'),
  file_names = c('file_a.csv', 'file_b.csv', 'file_c.txt')
)
original_tib
## # A tibble: 3 × 2
##   dir            file_names
##                  
## 1 some/path/blub file_a.csv
## 2 bla/here/      file_b.csv
## 3 direct/        file_c.txt

Here, assembling a path in the form directory/file_name.ext can be tricky. Some directories have trailing / and some don’t. So, working with paste0() or glue::glue() would be challenging. Thankfully, the path() function from the {fs} package doesn’t care whether trailing / are there or not.

original_tib |>
  mutate(path = path(dir, file_names))
## # A tibble: 3 × 3
##   dir            file_names path                     
##                                  
## 1 some/path/blub file_a.csv some/path/blub/file_a.csv
## 2 bla/here/      file_b.csv bla/here/file_b.csv      
## 3 direct/        file_c.txt direct/file_c.txt

Remove and set extensions

We can even modify file extensions really easily. That’s convenient when we want to take input from csv-files and then turn the data into images using the same file names.

original_tib |>
  mutate(
    path = path(dir, file_names),
    out_path = path_ext_set(path, 'png')
  )
## # A tibble: 3 × 4
##   dir            file_names path                      out_path                 
##                                                  
## 1 some/path/blub file_a.csv some/path/blub/file_a.csv some/path/blub/file_a.png
## 2 bla/here/      file_b.csv bla/here/file_b.csv       bla/here/file_b.png      
## 3 direct/        file_c.txt direct/file_c.txt         direct/file_c.png

Get directory infos

You can get information on a directory as a tree in the console. Here, I’m using a directory called raw-input inside my working directory to demonstrate that.

dir_tree('raw-input')
## raw-input
## ├── a
## │   └── dat.csv
## ├── b
## │   └── dat.csv
## └── c
##     └── dat.csv

You can also get lots of information on these files.

dir_info('raw-input')
## # A tibble: 3 × 18
##   path        type    size permissions modification_time   user  group device_id
##                             
## 1 raw-input/a direc…    4K rwxrwxr-x   2025-03-29 09:02:24 albe… albe…     66307
## 2 raw-input/b direc…    4K rwxrwxr-x   2025-03-29 09:04:33 albe… albe…     66307
## 3 raw-input/c direc…    4K rwxrwxr-x   2025-03-29 09:04:35 albe… albe…     66307
## # ℹ 10 more variables: hard_links , special_device_id , inode ,
## #   block_size , blocks , flags , generation ,
## #   access_time , change_time , birth_time

But in a lot of cases, it will probably suffice to just get the file paths.

dir_ls('raw-input')
## raw-input/a raw-input/b raw-input/c

In this function, you’ll need to use recurse = TRUE, though, to go into nested structures.

dir_ls('raw-input', recurse = TRUE)
## raw-input/a         raw-input/a/dat.csv raw-input/b         raw-input/b/dat.csv 
## raw-input/c         raw-input/c/dat.csv

Iterate over file paths

Usually, you don’t want to stop after finding the desired paths. You usually want to iterate over them. For this, you can save the output of dir_ls() into a vector and iterate through it using the map() or walk() function. Here, the function I use inside of walk() will

load the data using the specified path,
create a ggplot from it, and
save the image.

The tricky thing here is that I do want to save the files in an output directory. It is supposed to have the same structure as the raw-input directory. That’s why I also need to create the necessary paths and directories for that inside the function.

csv_files <- dir_ls(
  'raw-input',
  recurse = TRUE,
  regexp = '\\.csv$'
)

csv_files |>
  walk(
    \(file_path) {
      plt <- read_csv(file_path) |>
        ggplot(aes(col_a, col_b)) +
        geom_point(size = 10, col = 'dodgerblue4')

      out_path <- file_path |>
        path_ext_set('.png') |>
        str_replace('^raw-input', 'output')

      dir_create(path_dir(out_path))
      ggsave(filename = out_path)
    }
  )
## Rows: 3 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (3): col_a, col_b, col_c
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## Saving 6 x 4 in image
## Rows: 3 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (3): col_a, col_b, col_c
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## Saving 6 x 4 in image
## Rows: 3 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (3): col_a, col_b, col_c
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
## Saving 6 x 4 in image

Splendid. This should have worked and you can now see the output directory and the plots in the file tree.

dir_tree()
## .
## ├── index.qmd
## ├── index.rmarkdown
## ├── output
## │   ├── a
## │   │   └── dat.png
## │   ├── b
## │   │   └── dat.png
## │   └── c
## │       └── dat.png
## └── raw-input
##     ├── a
##     │   └── dat.csv
##     ├── b
##     │   └── dat.csv
##     └── c
##         └── dat.csv

The difference between = and <- finally explained

Albert Rapp — Sat, 15 Mar 2025 23:00:00 GMT

In today’s blogpost, we take a look at one of the most questions about R: What’s the difference between <- and =? And as always, there’s a video version for you:

Similar assignment

Let’s first talk about the similarities of both operators. Obviously, both work as expected when you assign values to variable names.

a = 1
a
## [1] 1
a <- 2
a
## [1] 2

This even works in functions. Take this function, It always returns 5 whatever you stick into the argument a:

fct <- function(some_nmbr) {
  print(glue::glue('I received {some_nmbr}.'))
  5
}
fct(some_nmbr = 10)
## I received 10.
## [1] 5
fct(some_nmbr <- 10)
## I received 10.
## [1] 5

Boring function but have you seen how both = and <- worked the same? That’s why it’s easy to think that both operators are indeed the same. But that’s where different scoping comes in.

Scoping

The <- operator has a broader scope. It can create variables in the global environment when used inside functions, while = typically only assigns within the local function environment.

rm(some_nmbr) # make sure that some_nmbr doesn't currently 
              # exist in global env
fct(some_nmbr = 10)
## I received 10.
## [1] 5

Even though we assigned some_nmbr = inside the fct() the variable some_nmbr isn’t available once fct() completed.

some_nmbr
## Error in eval(expr, envir, enclos): object 'some_nmbr' not found

But watch what happens if we use <- instead of =:

fct(some_nmbr <- 10)
## I received 10.
## [1] 5
some_nmbr
## [1] 10

The function still tells us that it received 10 and returns 5. But this time the assignment some_nmbr <- also worked globally. That’s why we can access some_nmbr after fct() terminated.

Don’t believe me yet? Let’s try it again.

fct(some_nmbr <- 20)
## I received 20.
## [1] 5
some_nmbr
## [1] 20

See? Now, some_nmbr in the global environment is 20. Interestingly, this only takes effect, when the variable some_nmbr is actually used inside of fct(). Let’s create another function.

fct2 <- function(some_nmbr) {
  5
}

This function doesn’t actually use some_nmbr in this function body at all. Now watch what happens when we use the <- assignment again.

fct2(some_nmbr <- 30)
## [1] 5
some_nmbr
## [1] 20

some_nmbr still has the value from before and not the new one. Funny, isn’t it? Anyway, due to these possible side effects you might want to use = inside of function calls.

Powerup of `<-`

Now that we have seen that <- is in a sense more powerful than =, let me also mention another power-up. You see, you can make the <- into <<- and get an even more powerful operator. You can use the inside a function to define a variable outside the function in the global environment.

Check out this example.

power_fct <- function() {
  defined_inside <- 20
  5
}

This function first used the regular <- operator to define a variable called defined_inside. This variable cannot be accessed once power_fct() terminates.

power_fct()
## [1] 5
defined_inside
## Error in eval(expr, envir, enclos): object 'defined_inside' not found

But watch what happens when we make change <- to <<- inside of power_fct().

power_fct <- function() {
  defined_inside <<- 20
  5
}
power_fct()
## [1] 5
defined_inside
## [1] 20

Magical, isn’t it? This trick can be really handy sometimes but also quite dangerous. Make sure you use this trick only when you know what you want to do.

data.table vs dplyr: A Side-by-Side Comparison

Albert Rapp — Sat, 08 Mar 2025 23:00:00 GMT

In today’s blogpost I show you how to do common data cleaning operations via both {data.table} and {dplyr}. These are two fantastic frameworks inside the R ecosystem. As always, there’s also a video version available:

Get data

First, let us take a look at our data.

library(dplyr)
library(data.table)
ames <- modeldata::ames |>
  janitor::clean_names() |> 
  as_tibble() # this is already the case but for comparability done here again
ames
## # A tibble: 2,930 × 74
##    ms_sub_class           ms_zoning lot_frontage lot_area street alley lot_shape
##                                              
##  1 One_Story_1946_and_Ne… Resident…          141    31770 Pave   No_A… Slightly…
##  2 One_Story_1946_and_Ne… Resident…           80    11622 Pave   No_A… Regular  
##  3 One_Story_1946_and_Ne… Resident…           81    14267 Pave   No_A… Slightly…
##  4 One_Story_1946_and_Ne… Resident…           93    11160 Pave   No_A… Regular  
##  5 Two_Story_1946_and_Ne… Resident…           74    13830 Pave   No_A… Slightly…
##  6 Two_Story_1946_and_Ne… Resident…           78     9978 Pave   No_A… Slightly…
##  7 One_Story_PUD_1946_an… Resident…           41     4920 Pave   No_A… Regular  
##  8 One_Story_PUD_1946_an… Resident…           43     5005 Pave   No_A… Slightly…
##  9 One_Story_PUD_1946_an… Resident…           39     5389 Pave   No_A… Slightly…
## 10 Two_Story_1946_and_Ne… Resident…           60     7500 Pave   No_A… Regular  
## # ℹ 2,920 more rows
## # ℹ 67 more variables: land_contour , utilities , lot_config ,
## #   land_slope , neighborhood , condition_1 , condition_2 ,
## #   bldg_type , house_style , overall_cond , year_built ,
## #   year_remod_add , roof_style , roof_matl ,
## #   exterior_1st , exterior_2nd , mas_vnr_type ,
## #   mas_vnr_area , exter_cond , foundation , bsmt_cond , …

We will need to convert it to a data.table.

df_ames <- as.data.table(ames)
df_ames
##                              ms_sub_class                ms_zoning lot_frontage
##                                                               
##    1: One_Story_1946_and_Newer_All_Styles  Residential_Low_Density          141
##    2: One_Story_1946_and_Newer_All_Styles Residential_High_Density           80
##    3: One_Story_1946_and_Newer_All_Styles  Residential_Low_Density           81
##    4: One_Story_1946_and_Newer_All_Styles  Residential_Low_Density           93
##    5:            Two_Story_1946_and_Newer  Residential_Low_Density           74
##   ---                                                                          
## 2926:                 Split_or_Multilevel  Residential_Low_Density           37
## 2927: One_Story_1946_and_Newer_All_Styles  Residential_Low_Density            0
## 2928:                         Split_Foyer  Residential_Low_Density           62
## 2929: One_Story_1946_and_Newer_All_Styles  Residential_Low_Density           77
## 2930:            Two_Story_1946_and_Newer  Residential_Low_Density           74
##       lot_area street           alley          lot_shape land_contour utilities
##                                             
##    1:    31770   Pave No_Alley_Access Slightly_Irregular          Lvl    AllPub
##    2:    11622   Pave No_Alley_Access            Regular          Lvl    AllPub
##    3:    14267   Pave No_Alley_Access Slightly_Irregular          Lvl    AllPub
##    4:    11160   Pave No_Alley_Access            Regular          Lvl    AllPub
##    5:    13830   Pave No_Alley_Access Slightly_Irregular          Lvl    AllPub
##   ---                                                                          
## 2926:     7937   Pave No_Alley_Access Slightly_Irregular          Lvl    AllPub
## 2927:     8885   Pave No_Alley_Access Slightly_Irregular          Low    AllPub
## 2928:    10441   Pave No_Alley_Access            Regular          Lvl    AllPub
## 2929:    10010   Pave No_Alley_Access            Regular          Lvl    AllPub
## 2930:     9627   Pave No_Alley_Access            Regular          Lvl    AllPub
##       lot_config land_slope neighborhood condition_1 condition_2 bldg_type
##                                       
##    1:     Corner        Gtl   North_Ames        Norm        Norm    OneFam
##    2:     Inside        Gtl   North_Ames       Feedr        Norm    OneFam
##    3:     Corner        Gtl   North_Ames        Norm        Norm    OneFam
##    4:     Corner        Gtl   North_Ames        Norm        Norm    OneFam
##    5:     Inside        Gtl      Gilbert        Norm        Norm    OneFam
##   ---                                                                     
## 2926:    CulDSac        Gtl     Mitchell        Norm        Norm    OneFam
## 2927:     Inside        Mod     Mitchell        Norm        Norm    OneFam
## 2928:     Inside        Gtl     Mitchell        Norm        Norm    OneFam
## 2929:     Inside        Mod     Mitchell        Norm        Norm    OneFam
## 2930:     Inside        Mod     Mitchell        Norm        Norm    OneFam
##       house_style  overall_cond year_built year_remod_add roof_style roof_matl
##                                             
##    1:   One_Story       Average       1960           1960        Hip   CompShg
##    2:   One_Story Above_Average       1961           1961      Gable   CompShg
##    3:   One_Story Above_Average       1958           1958        Hip   CompShg
##    4:   One_Story       Average       1968           1968        Hip   CompShg
##    5:   Two_Story       Average       1997           1998      Gable   CompShg
##   ---                                                                         
## 2926:        SLvl Above_Average       1984           1984      Gable   CompShg
## 2927:   One_Story       Average       1983           1983      Gable   CompShg
## 2928:      SFoyer       Average       1992           1992      Gable   CompShg
## 2929:   One_Story       Average       1974           1975      Gable   CompShg
## 2930:   Two_Story       Average       1993           1994      Gable   CompShg
##       exterior_1st exterior_2nd mas_vnr_type mas_vnr_area exter_cond foundation
##                                             
##    1:      BrkFace      Plywood        Stone          112    Typical     CBlock
##    2:      VinylSd      VinylSd         None            0    Typical     CBlock
##    3:      Wd Sdng      Wd Sdng      BrkFace          108    Typical     CBlock
##    4:      BrkFace      BrkFace         None            0    Typical     CBlock
##    5:      VinylSd      VinylSd         None            0    Typical      PConc
##   ---                                                                          
## 2926:      HdBoard      HdBoard         None            0    Typical     CBlock
## 2927:      HdBoard      HdBoard         None            0    Typical     CBlock
## 2928:      HdBoard      Wd Shng         None            0    Typical      PConc
## 2929:      HdBoard      HdBoard         None            0    Typical     CBlock
## 2930:      HdBoard      HdBoard      BrkFace           94    Typical      PConc
##       bsmt_cond bsmt_exposure bsmt_fin_type_1 bsmt_fin_sf_1 bsmt_fin_type_2
##                                               
##    1:      Good            Gd             BLQ             2             Unf
##    2:   Typical            No             Rec             6             LwQ
##    3:   Typical            No             ALQ             1             Unf
##    4:   Typical            No             ALQ             1             Unf
##    5:   Typical            No             GLQ             3             Unf
##   ---                                                                      
## 2926:   Typical            Av             GLQ             3             Unf
## 2927:   Typical            Av             BLQ             2             ALQ
## 2928:   Typical            Av             GLQ             3             Unf
## 2929:   Typical            Av             ALQ             1             LwQ
## 2930:   Typical            Av             LwQ             4             Unf
##       bsmt_fin_sf_2 bsmt_unf_sf total_bsmt_sf heating heating_qc central_air
##                                            
##    1:             0         441          1080    GasA       Fair           Y
##    2:           144         270           882    GasA    Typical           Y
##    3:             0         406          1329    GasA    Typical           Y
##    4:             0        1045          2110    GasA  Excellent           Y
##    5:             0         137           928    GasA       Good           Y
##   ---                                                                       
## 2926:             0         184          1003    GasA    Typical           Y
## 2927:           324         239           864    GasA    Typical           Y
## 2928:             0         575           912    GasA    Typical           Y
## 2929:           123         195          1389    GasA       Good           Y
## 2930:             0         238           996    GasA  Excellent           Y
##       electrical first_flr_sf second_flr_sf gr_liv_area bsmt_full_bath
##                                             
##    1:      SBrkr         1656             0        1656              1
##    2:      SBrkr          896             0         896              0
##    3:      SBrkr         1329             0        1329              0
##    4:      SBrkr         2110             0        2110              1
##    5:      SBrkr          928           701        1629              0
##   ---                                                                 
## 2926:      SBrkr         1003             0        1003              1
## 2927:      SBrkr          902             0         902              1
## 2928:      SBrkr          970             0         970              0
## 2929:      SBrkr         1389             0        1389              1
## 2930:      SBrkr          996          1004        2000              0
##       bsmt_half_bath full_bath half_bath bedroom_abv_gr kitchen_abv_gr
##                                              
##    1:              0         1         0              3              1
##    2:              0         1         0              2              1
##    3:              0         1         1              3              1
##    4:              0         2         1              3              1
##    5:              0         2         1              3              1
##   ---                                                                 
## 2926:              0         1         0              3              1
## 2927:              0         1         0              2              1
## 2928:              1         1         0              3              1
## 2929:              0         1         0              2              1
## 2930:              0         2         1              3              1
##       tot_rms_abv_grd functional fireplaces garage_type garage_finish
##                                          
##    1:               7        Typ          2      Attchd           Fin
##    2:               5        Typ          0      Attchd           Unf
##    3:               6        Typ          0      Attchd           Unf
##    4:               8        Typ          2      Attchd           Fin
##    5:               6        Typ          1      Attchd           Fin
##   ---                                                                
## 2926:               6        Typ          0      Detchd           Unf
## 2927:               5        Typ          0      Attchd           Unf
## 2928:               6        Typ          0   No_Garage     No_Garage
## 2929:               6        Typ          1      Attchd           RFn
## 2930:               9        Typ          1      Attchd           Fin
##       garage_cars garage_area garage_cond      paved_drive wood_deck_sf
##                                             
##    1:           2         528     Typical Partial_Pavement          210
##    2:           1         730     Typical            Paved          140
##    3:           1         312     Typical            Paved          393
##    4:           2         522     Typical            Paved            0
##    5:           2         482     Typical            Paved          212
##   ---                                                                  
## 2926:           2         588     Typical            Paved          120
## 2927:           2         484     Typical            Paved          164
## 2928:           0           0   No_Garage            Paved           80
## 2929:           2         418     Typical            Paved          240
## 2930:           3         650     Typical            Paved          190
##       open_porch_sf enclosed_porch three_season_porch screen_porch pool_area
##                                                    
##    1:            62              0                  0            0         0
##    2:             0              0                  0          120         0
##    3:            36              0                  0            0         0
##    4:             0              0                  0            0         0
##    5:            34              0                  0            0         0
##   ---                                                                       
## 2926:             0              0                  0            0         0
## 2927:             0              0                  0            0         0
## 2928:            32              0                  0            0         0
## 2929:            38              0                  0            0         0
## 2930:            48              0                  0            0         0
##       pool_qc           fence misc_feature misc_val mo_sold year_sold sale_type
##                                         
##    1: No_Pool        No_Fence         None        0       5      2010       WD 
##    2: No_Pool Minimum_Privacy         None        0       6      2010       WD 
##    3: No_Pool        No_Fence         Gar2    12500       6      2010       WD 
##    4: No_Pool        No_Fence         None        0       4      2010       WD 
##    5: No_Pool Minimum_Privacy         None        0       3      2010       WD 
##   ---                                                                          
## 2926: No_Pool    Good_Privacy         None        0       3      2006       WD 
## 2927: No_Pool Minimum_Privacy         None        0       6      2006       WD 
## 2928: No_Pool Minimum_Privacy         Shed      700       7      2006       WD 
## 2929: No_Pool        No_Fence         None        0       4      2006       WD 
## 2930: No_Pool        No_Fence         None        0      11      2006       WD 
##       sale_condition sale_price longitude latitude
##                              
##    1:         Normal     215000 -93.61975 42.05403
##    2:         Normal     105000 -93.61976 42.05301
##    3:         Normal     172000 -93.61939 42.05266
##    4:         Normal     244000 -93.61732 42.05125
##    5:         Normal     189900 -93.63893 42.06090
##   ---                                             
## 2926:         Normal     142500 -93.60478 41.98896
## 2927:         Normal     131000 -93.60268 41.98831
## 2928:         Normal     132000 -93.60685 41.98651
## 2929:         Normal     170000 -93.60019 41.99092
## 2930:         Normal     188000 -93.60000 41.98927

Also, let us set options for nicer data.table outputs. I don’t want lot’s of columns and rows to flood my console.

# Compact printing similar to tibble
options(
    datatable.print.topn = 5,     # Show only top 10 rows
    datatable.print.nrows = 10,   # Limit number of printed rows
    datatable.print.class = TRUE, # Show column classes
    datatable.print.trunc.cols = TRUE # Truncate wide columns
)

The general difference

{dplyr} uses functions/verbs that are chained together. This has the advantage that it’s easily readable to anybody because the function names are usually understandable even to non-programmers.

On the other hand {data.table} uses a super concise code style with lots of abbreviations. This has the advantage that the code is short but if you don’t know the system then you might feel lost.

So let me explain the system behind {data.table}. The most basic thing to understand is that a data.table df can be modified via a bracket using up to three different components. This might look something like df[i, j, by].

The first component i refers to rows that are “modified”. The second component j refers to columns that might be modified. And the third component by refers to any grouping you might want to use.

But enough theory. Let’s see this in action.

ames |> arrange(lot_frontage) 
## # A tibble: 2,930 × 74
##    ms_sub_class           ms_zoning lot_frontage lot_area street alley lot_shape
##                                              
##  1 One_Story_1946_and_Ne… Resident…            0     7980 Pave   No_A… Slightly…
##  2 One_Story_PUD_1946_an… Resident…            0     6820 Pave   No_A… Slightly…
##  3 Two_Story_1946_and_Ne… Floating…            0     7500 Pave   No_A… Regular  
##  4 One_Story_1946_and_Ne… Resident…            0    11241 Pave   No_A… Slightly…
##  5 One_Story_1946_and_Ne… Resident…            0    12537 Pave   No_A… Slightly…
##  6 Two_Story_1946_and_Ne… Resident…            0     7851 Pave   No_A… Regular  
##  7 Split_or_Multilevel    Resident…            0     7750 Pave   No_A… Slightly…
##  8 Two_Story_1946_and_Ne… Resident…            0     9505 Pave   No_A… Slightly…
##  9 Two_Story_1946_and_Ne… Resident…            0     8880 Pave   No_A… Slightly…
## 10 Two_Story_1946_and_Ne… Resident…            0     9453 Pave   No_A… Slightly…
## # ℹ 2,920 more rows
## # ℹ 67 more variables: land_contour , utilities , lot_config ,
## #   land_slope , neighborhood , condition_1 , condition_2 ,
## #   bldg_type , house_style , overall_cond , year_built ,
## #   year_remod_add , roof_style , roof_matl ,
## #   exterior_1st , exterior_2nd , mas_vnr_type ,
## #   mas_vnr_area , exter_cond , foundation , bsmt_cond , …

df_ames[order(lot_frontage)]
##                                    ms_sub_class                    ms_zoning
##                                                                 
##    1:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
##    2:              One_Story_PUD_1946_and_Newer      Residential_Low_Density
##    3:                  Two_Story_1946_and_Newer Floating_Village_Residential
##    4:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
##    5:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
##   ---                                                                       
## 2926:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
## 2927: Two_Family_conversion_All_Styles_and_Ages      Residential_Low_Density
## 2928:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
## 2929:       One_Story_1946_and_Newer_All_Styles      Residential_Low_Density
## 2930:                  Two_Story_1946_and_Newer      Residential_Low_Density
## 72 variable(s) not shown: [lot_frontage , lot_area , street , alley , lot_shape , land_contour , utilities , lot_config , land_slope , neighborhood , ...]

Filter for specific rows

dplyr
data.table

ames |> filter(sale_price > 300000)
## # A tibble: 230 × 74
##    ms_sub_class           ms_zoning lot_frontage lot_area street alley lot_shape
##                                              
##  1 Two_Story_1946_and_Ne… Resident…           47    53504 Pave   No_A… Moderate…
##  2 One_Story_1946_and_Ne… Resident…           88    11394 Pave   No_A… Regular  
##  3 Two_Story_1946_and_Ne… Resident…          102    12858 Pave   No_A… Slightly…
##  4 One_Story_1946_and_Ne… Resident…           98    11478 Pave   No_A… Regular  
##  5 One_Story_1946_and_Ne… Resident…           83    10159 Pave   No_A… Slightly…
##  6 One_Story_1946_and_Ne… Resident…          100    12919 Pave   No_A… Slightly…
##  7 One_Story_1946_and_Ne… Resident…          110    14300 Pave   No_A… Regular  
##  8 Two_Story_1946_and_Ne… Resident…          105    13650 Pave   No_A… Regular  
##  9 One_Story_PUD_1946_an… Resident…           61     7658 Pave   No_A… Regular  
## 10 Two_Story_1946_and_Ne… Resident…          108    14774 Pave   No_A… Slightly…
## # ℹ 220 more rows
## # ℹ 67 more variables: land_contour , utilities , lot_config ,
## #   land_slope , neighborhood , condition_1 , condition_2 ,
## #   bldg_type , house_style , overall_cond , year_built ,
## #   year_remod_add , roof_style , roof_matl ,
## #   exterior_1st , exterior_2nd , mas_vnr_type ,
## #   mas_vnr_area , exter_cond , foundation , bsmt_cond , …

df_ames[sale_price > 300000]
##                             ms_sub_class               ms_zoning lot_frontage
##                                                             
##   1:            Two_Story_1946_and_Newer Residential_Low_Density           47
##   2: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           88
##   3:            Two_Story_1946_and_Newer Residential_Low_Density          102
##   4: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           98
##   5: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           83
##  ---                                                                         
## 226:            Two_Story_1946_and_Newer Residential_Low_Density           42
## 227: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           85
## 228: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           95
## 229: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           88
## 230: One_Story_1946_and_Newer_All_Styles Residential_Low_Density           88
## 71 variable(s) not shown: [lot_area , street , alley , lot_shape , land_contour , utilities , lot_config , land_slope , neighborhood , condition_1 , ...]

Select specific columns

ames |> 
  filter(sale_price > 300000) |> 
  select(neighborhood, sale_price, lot_area)
## # A tibble: 230 × 3
##    neighborhood       sale_price lot_area
##                           
##  1 Stone_Brook            538000    53504
##  2 Stone_Brook            394432    11394
##  3 Northridge_Heights     376162    12858
##  4 Northridge_Heights     306000    11478
##  5 Northridge_Heights     395192    10159
##  6 Northridge_Heights     611657    12919
##  7 Northridge_Heights     500000    14300
##  8 Northridge_Heights     320000    13650
##  9 Northridge_Heights     319900     7658
## 10 Northridge             333168    14774
## # ℹ 220 more rows

df_ames[
  sale_price > 300000, 
  list(neighborhood, sale_price, lot_area)
]
##            neighborhood sale_price lot_area
##                            
##   1:        Stone_Brook     538000    53504
##   2:        Stone_Brook     394432    11394
##   3: Northridge_Heights     376162    12858
##   4: Northridge_Heights     306000    11478
##   5: Northridge_Heights     395192    10159
##  ---                                       
## 226:         Timberland     335000    26178
## 227:         Timberland     312500    14331
## 228:         Timberland     320000    13618
## 229:         Timberland     369900    11443
## 230:         Timberland     359900    11577

If you don’t want to type out the list() command, you can also use data.table’s shorthand notation .().

df_ames[
  sale_price > 300000, 
  .(neighborhood, sale_price, lot_area)
]
##            neighborhood sale_price lot_area
##                            
##   1:        Stone_Brook     538000    53504
##   2:        Stone_Brook     394432    11394
##   3: Northridge_Heights     376162    12858
##   4: Northridge_Heights     306000    11478
##   5: Northridge_Heights     395192    10159
##  ---                                       
## 226:         Timberland     335000    26178
## 227:         Timberland     312500    14331
## 228:         Timberland     320000    13618
## 229:         Timberland     369900    11443
## 230:         Timberland     359900    11577

Select many columns by name

dplyr
data.table

ames |> select(contains('area'))
## # A tibble: 2,930 × 5
##    lot_area mas_vnr_area gr_liv_area garage_area pool_area
##                                  
##  1    31770          112        1656         528         0
##  2    11622            0         896         730         0
##  3    14267          108        1329         312         0
##  4    11160            0        2110         522         0
##  5    13830            0        1629         482         0
##  6     9978           20        1604         470         0
##  7     4920            0        1338         582         0
##  8     5005            0        1280         506         0
##  9     5389            0        1616         608         0
## 10     7500            0        1804         442         0
## # ℹ 2,920 more rows

df_ames[, .SD, .SDcols = patterns('area')]
##       lot_area mas_vnr_area gr_liv_area garage_area pool_area
##                                     
##    1:    31770          112        1656         528         0
##    2:    11622            0         896         730         0
##    3:    14267          108        1329         312         0
##    4:    11160            0        2110         522         0
##    5:    13830            0        1629         482         0
##   ---                                                        
## 2926:     7937            0        1003         588         0
## 2927:     8885            0         902         484         0
## 2928:    10441            0         970           0         0
## 2929:    10010            0        1389         418         0
## 2930:     9627           94        2000         650         0

Select many columns by type

dplyr
data.table

ames |> 
  select(where(is.numeric))
## # A tibble: 2,930 × 34
##    lot_frontage lot_area year_built year_remod_add mas_vnr_area bsmt_fin_sf_1
##                                                
##  1          141    31770       1960           1960          112             2
##  2           80    11622       1961           1961            0             6
##  3           81    14267       1958           1958          108             1
##  4           93    11160       1968           1968            0             1
##  5           74    13830       1997           1998            0             3
##  6           78     9978       1998           1998           20             3
##  7           41     4920       2001           2001            0             3
##  8           43     5005       1992           1992            0             1
##  9           39     5389       1995           1996            0             3
## 10           60     7500       1999           1999            0             7
## # ℹ 2,920 more rows
## # ℹ 28 more variables: bsmt_fin_sf_2 , bsmt_unf_sf ,
## #   total_bsmt_sf , first_flr_sf , second_flr_sf ,
## #   gr_liv_area , bsmt_full_bath , bsmt_half_bath ,
## #   full_bath , half_bath , bedroom_abv_gr ,
## #   kitchen_abv_gr , tot_rms_abv_grd , fireplaces ,
## #   garage_cars , garage_area , wood_deck_sf , …

df_ames[, .SD, .SDcols = is.numeric]
##       lot_frontage lot_area year_built year_remod_add mas_vnr_area
##                                          
##    1:          141    31770       1960           1960          112
##    2:           80    11622       1961           1961            0
##    3:           81    14267       1958           1958          108
##    4:           93    11160       1968           1968            0
##    5:           74    13830       1997           1998            0
##   ---                                                             
## 2926:           37     7937       1984           1984            0
## 2927:            0     8885       1983           1983            0
## 2928:           62    10441       1992           1992            0
## 2929:           77    10010       1974           1975            0
## 2930:           74     9627       1993           1994           94
## 29 variable(s) not shown: [bsmt_fin_sf_1 , bsmt_fin_sf_2 , bsmt_unf_sf , total_bsmt_sf , first_flr_sf , second_flr_sf , gr_liv_area , bsmt_full_bath , bsmt_half_bath , full_bath , ...]

Compute a new column

dplyr
data.table (new data.table)

ames |> 
  filter(sale_price > 300000) |> 
  select(neighborhood, sale_price, lot_area) |> 
  mutate(
    price_by_lot_area = sale_price / lot_area
  )
## # A tibble: 230 × 4
##    neighborhood       sale_price lot_area price_by_lot_area
##                                        
##  1 Stone_Brook            538000    53504              10.1
##  2 Stone_Brook            394432    11394              34.6
##  3 Northridge_Heights     376162    12858              29.3
##  4 Northridge_Heights     306000    11478              26.7
##  5 Northridge_Heights     395192    10159              38.9
##  6 Northridge_Heights     611657    12919              47.3
##  7 Northridge_Heights     500000    14300              35.0
##  8 Northridge_Heights     320000    13650              23.4
##  9 Northridge_Heights     319900     7658              41.8
## 10 Northridge             333168    14774              22.6
## # ℹ 220 more rows

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area
  )
]
##            neighborhood sale_price lot_area price_by_lot_area
##                                         
##   1:        Stone_Brook     538000    53504          10.05532
##   2:        Stone_Brook     394432    11394          34.61752
##   3: Northridge_Heights     376162    12858          29.25509
##   4: Northridge_Heights     306000    11478          26.65970
##   5: Northridge_Heights     395192    10159          38.90068
##  ---                                                         
## 226:         Timberland     335000    26178          12.79701
## 227:         Timberland     312500    14331          21.80588
## 228:         Timberland     320000    13618          23.49831
## 229:         Timberland     369900    11443          32.32544
## 230:         Timberland     359900    11577          31.08750

Specialty of `data.table`: modify in place

In the last step we have seen that mutate() and using a list() inside the [] of a data.table work almost similarly. But there’s a specialty that you might want to keep in mind.

You see, in the previous example we created a new data.table. And that’s why we have seen an output after df_ames[...]. But checkout what happens if I don’t use list() and instead use the := operator inside the brackets.

df_ames[
  sale_price > 300000, 
  price_by_lot_area := sale_price / lot_area
]

See? No output. But if we look at df_ames, we’ll notice that the price_by_lot_area was actually created. And that happened even though we never overwrote df_ames like df_ames <-.

# Notice no filter on rows here as first argument was left blank.
df_ames[, price_by_lot_area] |> head(30)
##  [1]       NA       NA       NA       NA       NA       NA       NA       NA
##  [9]       NA       NA       NA       NA       NA       NA       NA 10.05532
## [17]       NA 34.61752       NA       NA       NA       NA       NA       NA
## [25]       NA       NA       NA       NA       NA       NA

Notice now that we have lots of NAs in that column now. This happens because we instructed R to only compute the price_by_lot_area column where sale_price > 300000. Interesting how that works, isn’t it?

If you wanted to calculate all values, you could leave out the filtering condition.

df_ames[, price_by_lot_area := sale_price / lot_area]
df_ames[, price_by_lot_area] |> head(30)
##  [1]  6.767391  9.034590 12.055793 21.863799 13.731020 19.593105 43.394309
##  [8] 38.261738 43.885693 25.200000 17.590000 23.182957 21.471078 16.853381
## [15] 31.085044 10.055323 13.515741 34.617518  7.367541 15.939279 16.168837
## [22] 16.000000 28.800000 13.255048 11.956608 16.804734 15.000000 10.952381
## [29] 31.410038 57.142857

Calculate multiple new columns

ames |> 
  filter(sale_price > 300000) |> 
  select(neighborhood, sale_price, lot_area) |> 
  mutate(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log_sale_price / lot_area
  )
## # A tibble: 230 × 6
##    neighborhood       sale_price lot_area price_by_lot_area log_sale_price
##                                                  
##  1 Stone_Brook            538000    53504              10.1           13.2
##  2 Stone_Brook            394432    11394              34.6           12.9
##  3 Northridge_Heights     376162    12858              29.3           12.8
##  4 Northridge_Heights     306000    11478              26.7           12.6
##  5 Northridge_Heights     395192    10159              38.9           12.9
##  6 Northridge_Heights     611657    12919              47.3           13.3
##  7 Northridge_Heights     500000    14300              35.0           13.1
##  8 Northridge_Heights     320000    13650              23.4           12.7
##  9 Northridge_Heights     319900     7658              41.8           12.7
## 10 Northridge             333168    14774              22.6           12.7
## # ℹ 220 more rows
## # ℹ 1 more variable: log_price_by_lot_area

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log(sale_price) / lot_area
  )
]
##            neighborhood sale_price lot_area price_by_lot_area log_sale_price
##                                                   
##   1:        Stone_Brook     538000    53504          10.05532       13.19561
##   2:        Stone_Brook     394432    11394          34.61752       12.88520
##   3: Northridge_Heights     376162    12858          29.25509       12.83778
##   4: Northridge_Heights     306000    11478          26.65970       12.63134
##   5: Northridge_Heights     395192    10159          38.90068       12.88713
##  ---                                                                        
## 226:         Timberland     335000    26178          12.79701       12.72189
## 227:         Timberland     312500    14331          21.80588       12.65236
## 228:         Timberland     320000    13618          23.49831       12.67608
## 229:         Timberland     369900    11443          32.32544       12.82099
## 230:         Timberland     359900    11577          31.08750       12.79358
## 1 variable(s) not shown: [log_price_by_lot_area ]

To create multiple columns in place, you can either use the let() command

df_ames[
  sale_price > 300000, 
  let(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log(sale_price) / lot_area
  )
]

or the := operator.

df_ames[
  sale_price > 300000, 
  `:=`(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log(sale_price) / lot_area
  )
]

Specialty of `data.table`: chaining

Notice that in both previous data.table examples we have used log(sale_price) twice. That means after calculating the log_sale_price column, we didn’t actually use that column later on. The reason for this is that it simply doesn’t work. You cannot calculate a new column and use it in the same step.

# reset df_ames so that `log_sale_price` doesn't exist
# due to previous calculations.
df_ames <- as.data.table(ames) 
df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log_sale_price / lot_area # log_sale_price here
  )
]
## Error in eval(jsub, SDenv, parent.frame()): object 'log_sale_price' not found

One way to fix that is to use chain operations. You can do so by using multiple brackets like df_ames[...][...], you can chain multiple operations.

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
][, log_price_by_lot_area := log_sale_price / lot_area]

Notice that this technically worked and didn’t throw an error, but you cannot find any of the new columns in df_ames.

df_ames[, list(neighborhood, log_price_by_lot_area)]
## Error in eval(jsub, SDenv, parent.frame()): object 'log_price_by_lot_area' not found
df_ames[, list(neighborhood, log_sale_price)]
## Error in eval(jsub, SDenv, parent.frame()): object 'log_sale_price' not found

Here’s why that happened: First, we created a new data.table by not using the in-place version. Remember how the list() command creates a new data.table? That’s why you get an output here:

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
]
##            neighborhood sale_price lot_area price_by_lot_area log_sale_price
##                                                   
##   1:        Stone_Brook     538000    53504          10.05532       13.19561
##   2:        Stone_Brook     394432    11394          34.61752       12.88520
##   3: Northridge_Heights     376162    12858          29.25509       12.83778
##   4: Northridge_Heights     306000    11478          26.65970       12.63134
##   5: Northridge_Heights     395192    10159          38.90068       12.88713
##  ---                                                                        
## 226:         Timberland     335000    26178          12.79701       12.72189
## 227:         Timberland     312500    14331          21.80588       12.65236
## 228:         Timberland     320000    13618          23.49831       12.67608
## 229:         Timberland     369900    11443          32.32544       12.82099
## 230:         Timberland     359900    11577          31.08750       12.79358

But if you don’t save this into a new variable, then this new data table isn’t saved. Now, in the next step, we use the in-place mode of operations on the new data.table.

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
][, log_price_by_lot_area := log_sale_price / lot_area]

This technically works but in-place calculations don’t return anything. They just modify the existing object. But the existing object is the new data.table that we still never saved anywhere. Hence, that calculation runs successfully but the results vanish the moment the calculation finishes.

What’s worse is that even if you try to save these results later, it’s no use. See? No results here:

df_test <- df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
][, log_price_by_lot_area := log_sale_price / lot_area]
df_test

So there are two ways you could fix this. Either chain your results without using the in-place calculations. This means either using list() or .() in later steps of the chain.

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
][, .(log_price_by_lot_area = log_sale_price / lot_area)]
##      log_price_by_lot_area
##                      
##   1:          0.0002466285
##   2:          0.0011308761
##   3:          0.0009984271
##   4:          0.0011004827
##   5:          0.0012685429
##  ---                      
## 226:          0.0004859762
## 227:          0.0008828665
## 228:          0.0009308324
## 229:          0.0011204219
## 230:          0.0011050861

As we’ve seen before, this will then only return a data.table with the filtered rows. Or alternatively, you could use only in-place calculation.

df_ames[
  sale_price > 300000, 
  let(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
][, log_price_by_lot_area := log_sale_price / lot_area]

df_ames[
  # Set filter to see both NA and non-NA values
  sale_price > 200000, 
  .(neighborhood, price_by_lot_area, log_sale_price, log_price_by_lot_area)
]
##      neighborhood price_by_lot_area log_sale_price log_price_by_lot_area
##                                                    
##   1:   North_Ames                NA             NA                    NA
##   2:   North_Ames                NA             NA                    NA
##   3:  Stone_Brook                NA             NA                    NA
##   4:  Stone_Brook                NA             NA                    NA
##   5:  Stone_Brook                NA             NA                    NA
##  ---                                                                    
## 853:   Timberland          21.80588       12.65236          0.0008828665
## 854:   Timberland          23.49831       12.67608          0.0009308324
## 855:   Timberland          32.32544       12.82099          0.0011204219
## 856:   Timberland          31.08750       12.79358          0.0011050861
## 857:     Mitchell                NA             NA                    NA

But personally I prefer to chain things with a pipe operator. With the R native pipe operator |> and its placeholder variable _ this works pretty well.

df_ames[
  sale_price > 300000, 
  list(
    neighborhood, 
    sale_price, 
    lot_area,
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price)
  )
] |> 
  _[, .(log_price_by_lot_area = log_sale_price / lot_area)]
##      log_price_by_lot_area
##                      
##   1:          0.0002466285
##   2:          0.0011308761
##   3:          0.0009984271
##   4:          0.0011004827
##   5:          0.0012685429
##  ---                      
## 226:          0.0004859762
## 227:          0.0008828665
## 228:          0.0009308324
## 229:          0.0011204219
## 230:          0.0011050861

Calculate summary statistics (without grouping)

dplyr
data.table

ames |> 
  filter(sale_price > 300000) |> 
  select(neighborhood, sale_price, lot_area) |> 
  mutate(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log_sale_price / lot_area
  ) |> 
  summarize(
    mean_sale_price = mean(sale_price),
    n_houses = n()
  )
## # A tibble: 1 × 2
##   mean_sale_price n_houses
##                 
## 1         378693.      230

In data.tables the special variable .N refers to the amount of rows. In this case .N refers to the amount of all rows. In a grouped setting, .N would refer to the group sizes.

df_ames[
  sale_price > 300000, 
  list(
    mean_sale_price = mean(sale_price),
    n_houses = .N
  )
]
##    mean_sale_price n_houses
##                  
## 1:        378693.4      230

Calculate summary statistics (with grouping)

dplyr
data.table

ames |> 
  filter(sale_price > 300000) |> 
  select(neighborhood, sale_price, lot_area) |> 
  mutate(
    price_by_lot_area = sale_price / lot_area,
    log_sale_price = log(sale_price),
    log_price_by_lot_area = log_sale_price / lot_area
  ) |> 
  summarize(
    mean_sale_price = mean(sale_price),
    n_houses = n(),
    .by = neighborhood # use grouping
  )
## # A tibble: 16 × 3
##    neighborhood       mean_sale_price n_houses
##                                
##  1 Stone_Brook                417381.       27
##  2 Northridge_Heights         389422.       91
##  3 Northridge                 387282.       37
##  4 Clear_Creek                315000         2
##  5 Edwards                    347677.        3
##  6 Sawyer_West                312533.        3
##  7 College_Creek              363104.       10
##  8 Crawford                   347410        10
##  9 Timberland                 353320        15
## 10 North_Ames                 323300         2
## 11 Gilbert                    335500         3
## 12 Somerset                   345156        19
## 13 Veenker                    350188.        4
## 14 Northwest_Ames             306000         1
## 15 Old_Town                   400000         2
## 16 Green_Hills                330000         1

In data.tables the special variable .N refers to the amount of rows. In this case .N refers to the amount of all rows. In a grouped setting, .N would refer to the group sizes.

df_ames[
  sale_price > 300000, 
  list(
    mean_sale_price = mean(sale_price),
    n_houses = .N
  ),
  by = neighborhood  # use grouping
]
##           neighborhood mean_sale_price n_houses
##                                
##  1:        Stone_Brook        417381.1       27
##  2: Northridge_Heights        389421.9       91
##  3:         Northridge        387281.6       37
##  4:        Clear_Creek        315000.0        2
##  5:            Edwards        347676.7        3
## ---                                            
## 12:           Somerset        345156.0       19
## 13:            Veenker        350187.5        4
## 14:     Northwest_Ames        306000.0        1
## 15:           Old_Town        400000.0        2
## 16:        Green_Hills        330000.0        1

Iterate over multiple columns

Let’s combine our knowledge of selecting multiple columns to calculate summary stats for many columns.

dplyr
data.table

In dplyr this needs the across() function inside of summarize().

ames |> 
  summarize(
    across(
      .cols = contains('area'),
      .fns = mean
    ),
    .by = neighborhood
  )
## # A tibble: 28 × 6
##    neighborhood        lot_area mas_vnr_area gr_liv_area garage_area pool_area
##                                                 
##  1 North_Ames            10040.         92.2       1292.        423.     1.99 
##  2 Gilbert               11342.         39.2       1621.        454.     0    
##  3 Stone_Brook           11383.        211.        1949.        626.     0    
##  4 Northwest_Ames        11662.        171.        1689.        535.     8.91 
##  5 Somerset               7611.        138.        1605.        620.     0    
##  6 Briardale              1841.        364.        1115.        292      0    
##  7 Northpark_Villa        2756.          0         1244.        421.     0    
##  8 Northridge_Heights    11217.        331.        1943.        748.     0.867
##  9 Bloomington_Heights    3399.         41.6       1405.        493.     0    
## 10 Northridge            12976.        417.        2481.        721.     7.82 
## # ℹ 18 more rows

In data.table, you can just use the .SD variable again and combine that with lapply().

df_ames[, 
  c(
    lapply(.SD, mean, na.rm = TRUE),
    n_houses = .N
  ),
  by = neighborhood,
  .SDcols = patterns('area')
]
##       neighborhood  lot_area mas_vnr_area gr_liv_area garage_area pool_area
##                                             
##  1:     North_Ames 10039.702     92.24605    1292.054    423.2460  1.986456
##  2:        Gilbert 11342.370     39.17576    1620.897    453.8000  0.000000
##  3:    Stone_Brook 11383.392    211.15686    1949.216    625.7059  0.000000
##  4: Northwest_Ames 11661.695    170.78626    1688.771    534.9084  8.908397
##  5:       Somerset  7610.676    137.56044    1604.830    620.3407  0.000000
## ---                                                                        
## 24:     Timberland 18233.542    179.87500    1714.639    604.6667  0.000000
## 25: Meadow_Village  2109.027      7.27027    1066.703    226.3784  0.000000
## 26:        Veenker 15482.208    139.00000    1819.542    574.0833 33.333333
## 27:    Green_Hills  9001.000    119.50000    1398.500    299.0000  0.000000
## 28:       Landmark  3612.000      0.00000    1320.000    484.0000  0.000000
## 3 variable(s) not shown: [price_by_lot_area , log_price_by_lot_area , N ]

Notice that for data.table we had to use the c() function to comine results. Had we used a list via .(), then results wouldn’t look quite the same.

df_ames[, 
  .(
    lapply(.SD, mean, na.rm = TRUE),
    n_houses = .N
  ),
  by = neighborhood,
  .SDcols = patterns('area')
]
##      neighborhood       V1 n_houses
##                   
##   1:   North_Ames  10039.7      443
##   2:   North_Ames 92.24605      443
##   3:   North_Ames 1292.054      443
##   4:   North_Ames  423.246      443
##   5:   North_Ames 1.986456      443
##  ---                               
## 192:     Landmark     1320        1
## 193:     Landmark      484        1
## 194:     Landmark        0        1
## 195:     Landmark      NaN        1
## 196:     Landmark      NaN        1

Or we could also cover, say, numeric columns. In that case we shouldn’t use .cols = contains() and .SDcols = patterns(). Instead we’d use .cols = where(is.numeric) and .SDcols = is.numeric:

dplyr
data.table

ames |> 
  summarize(
    across(
      .cols = where(is.numeric),
      .fns = mean
    ),
    .by = neighborhood
  )
## # A tibble: 28 × 35
##    neighborhood     lot_frontage lot_area year_built year_remod_add mas_vnr_area
##                                                   
##  1 North_Ames               63.7   10040.      1960.          1971.         92.2
##  2 Gilbert                  49.9   11342.      1998.          1999.         39.2
##  3 Stone_Brook              56.1   11383.      1999.          2000.        211. 
##  4 Northwest_Ames           52.9   11662.      1976.          1980.        171. 
##  5 Somerset                 57.5    7611.      2005.          2005.        138. 
##  6 Briardale                21.5    1841.      1972.          1973         364. 
##  7 Northpark_Villa          25.7    2756.      1976.          1976.          0  
##  8 Northridge_Heig…         82.7   11217.      2006.          2006.        331. 
##  9 Bloomington_Hei…         33.5    3399.      2005.          2005.         41.6
## 10 Northridge               69.7   12976.      1995.          1997.        417. 
## # ℹ 18 more rows
## # ℹ 29 more variables: bsmt_fin_sf_1 , bsmt_fin_sf_2 ,
## #   bsmt_unf_sf , total_bsmt_sf , first_flr_sf ,
## #   second_flr_sf , gr_liv_area , bsmt_full_bath ,
## #   bsmt_half_bath , full_bath , half_bath ,
## #   bedroom_abv_gr , kitchen_abv_gr , tot_rms_abv_grd ,
## #   fireplaces , garage_cars , garage_area , …

In data.tables the special variable .N refers to the amount of rows. In this case .N refers to the amount of all rows. In a grouped setting, .N would refer to the group sizes.

df_ames[, 
  c(
    lapply(.SD, mean, na.rm = TRUE),
    n_houses = .N
  ),
  by = neighborhood,
  .SDcols = is.numeric
]
##       neighborhood lot_frontage  lot_area year_built year_remod_add
##                                          
##  1:     North_Ames     63.66591 10039.702   1959.876       1970.652
##  2:        Gilbert     49.92121 11342.370   1998.303       1998.994
##  3:    Stone_Brook     56.07843 11383.392   1999.431       1999.725
##  4: Northwest_Ames     52.89313 11661.695   1975.588       1980.252
##  5:       Somerset     57.45604  7610.676   2004.621       2004.956
## ---                                                                
## 24:     Timberland     64.25000 18233.542   1995.333       1996.667
## 25: Meadow_Village     22.83784  2109.027   1972.243       1976.027
## 26:        Veenker     48.00000 15482.208   1981.708       1987.750
## 27:    Green_Hills      0.00000  9001.000   1992.000       1992.000
## 28:       Landmark      0.00000  3612.000   1993.000       1994.000
## 34 variable(s) not shown: [mas_vnr_area , bsmt_fin_sf_1 , bsmt_fin_sf_2 , bsmt_unf_sf , total_bsmt_sf , first_flr_sf , second_flr_sf , gr_liv_area , bsmt_full_bath , bsmt_half_bath , ...]

5 Levels of Data Wrangling Every R User Must Master

Albert Rapp — Sat, 15 Feb 2025 23:00:00 GMT

In today’s blog post, I’ll show you how to go from {dplyr} beginner to pro in 5 simple stages. As always there’s also a video version available to this blog post:

On this endeavor, let’s work with the Ames housing data set.

library(tidyverse)
housing_dat <- modeldata::ames
housing_dat
## # A tibble: 2,930 × 74
##    MS_SubClass            MS_Zoning Lot_Frontage Lot_Area Street Alley Lot_Shape
##  *                                           
##  1 One_Story_1946_and_Ne… Resident…          141    31770 Pave   No_A… Slightly…
##  2 One_Story_1946_and_Ne… Resident…           80    11622 Pave   No_A… Regular  
##  3 One_Story_1946_and_Ne… Resident…           81    14267 Pave   No_A… Slightly…
##  4 One_Story_1946_and_Ne… Resident…           93    11160 Pave   No_A… Regular  
##  5 Two_Story_1946_and_Ne… Resident…           74    13830 Pave   No_A… Slightly…
##  6 Two_Story_1946_and_Ne… Resident…           78     9978 Pave   No_A… Slightly…
##  7 One_Story_PUD_1946_an… Resident…           41     4920 Pave   No_A… Regular  
##  8 One_Story_PUD_1946_an… Resident…           43     5005 Pave   No_A… Slightly…
##  9 One_Story_PUD_1946_an… Resident…           39     5389 Pave   No_A… Slightly…
## 10 Two_Story_1946_and_Ne… Resident…           60     7500 Pave   No_A… Regular  
## # ℹ 2,920 more rows
## # ℹ 67 more variables: Land_Contour , Utilities , Lot_Config ,
## #   Land_Slope , Neighborhood , Condition_1 , Condition_2 ,
## #   Bldg_Type , House_Style , Overall_Cond , Year_Built ,
## #   Year_Remod_Add , Roof_Style , Roof_Matl ,
## #   Exterior_1st , Exterior_2nd , Mas_Vnr_Type ,
## #   Mas_Vnr_Area , Exter_Cond , Foundation , Bsmt_Cond , …

Level 1: Do everything manually

Let’s run a little analysis of comparing area sizes of different parts of the house by neighborhood. All of the columns that contain sizes are indicated by “SF” (as in square fee).

In our analysis, we’ll grab the required columns, transform them to square meters and then calculate the mean size by neighborhood. In Level 1, this will all be done manually:

housing_dat |> 
  select(
    Neighborhood,
    BsmtFin_SF_1,
    BsmtFin_SF_2,
    Bsmt_Unf_SF,
    Total_Bsmt_SF,
    First_Flr_SF,
    Second_Flr_SF,
    Wood_Deck_SF,
    Open_Porch_SF
  ) |> 
  mutate(
    BsmtFin_SF_1   = BsmtFin_SF_1 / 10.7639,
    BsmtFin_SF_2   = BsmtFin_SF_2 / 10.7639,
    Bsmt_Unf_SF    = Bsmt_Unf_SF / 10.7639,
    Total_Bsmt_SF  = Total_Bsmt_SF / 10.7639,
    First_Flr_SF   = First_Flr_SF / 10.7639,
    Second_Flr_SF  = Second_Flr_SF / 10.7639,
    Wood_Deck_SF   = Wood_Deck_SF / 10.7639,
    Open_Porch_SF  = Open_Porch_SF / 10.7639
  ) |> 
  summarize(
    BsmtFin_SF_1   = mean(BsmtFin_SF_1), 
    BsmtFin_SF_2   = mean(BsmtFin_SF_2), 
    Bsmt_Unf_SF    = mean(Bsmt_Unf_SF), 
    Total_Bsmt_SF  = mean(Total_Bsmt_SF), 
    First_Flr_SF   = mean(First_Flr_SF), 
    Second_Flr_SF  = mean(Second_Flr_SF), 
    Wood_Deck_SF   = mean(Wood_Deck_SF), 
    Open_Porch_SF  = mean(Open_Porch_SF),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

Level 2: Use tidyselect helpers in `select()`

In Level 2, we can replace the long list of names by a tidyselect helper inside the select() call.

housing_dat |> 
  select(Neighborhood, contains('SF')) |> 
  mutate(
    BsmtFin_SF_1   = BsmtFin_SF_1 / 10.7639,
    BsmtFin_SF_2   = BsmtFin_SF_2 / 10.7639,
    Bsmt_Unf_SF    = Bsmt_Unf_SF / 10.7639,
    Total_Bsmt_SF  = Total_Bsmt_SF / 10.7639,
    First_Flr_SF   = First_Flr_SF / 10.7639,
    Second_Flr_SF  = Second_Flr_SF / 10.7639,
    Wood_Deck_SF   = Wood_Deck_SF / 10.7639,
    Open_Porch_SF  = Open_Porch_SF / 10.7639
  ) |> 
  summarize(
    BsmtFin_SF_1   = mean(BsmtFin_SF_1), 
    BsmtFin_SF_2   = mean(BsmtFin_SF_2), 
    Bsmt_Unf_SF    = mean(Bsmt_Unf_SF), 
    Total_Bsmt_SF  = mean(Total_Bsmt_SF), 
    First_Flr_SF   = mean(First_Flr_SF), 
    Second_Flr_SF  = mean(Second_Flr_SF), 
    Wood_Deck_SF   = mean(Wood_Deck_SF), 
    Open_Porch_SF  = mean(Open_Porch_SF),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

Level 3: Use across with built-in functions

Now, we can remove a lot of duplicate code inside of summarize() with help from the across() function. Inside this helper function, we specify which columns (.cols) we want to iterate over and what function .fns we want to apply each time.

housing_dat |> 
  select(Neighborhood, contains('SF')) |> 
  mutate(
    BsmtFin_SF_1   = BsmtFin_SF_1 / 10.7639,
    BsmtFin_SF_2   = BsmtFin_SF_2 / 10.7639,
    Bsmt_Unf_SF    = Bsmt_Unf_SF / 10.7639,
    Total_Bsmt_SF  = Total_Bsmt_SF / 10.7639,
    First_Flr_SF   = First_Flr_SF / 10.7639,
    Second_Flr_SF  = Second_Flr_SF / 10.7639,
    Wood_Deck_SF   = Wood_Deck_SF / 10.7639,
    Open_Porch_SF  = Open_Porch_SF / 10.7639
  ) |> 
  summarize(
    across(
      .cols = c(
        BsmtFin_SF_1,
        BsmtFin_SF_2,
        Bsmt_Unf_SF,
        Total_Bsmt_SF,
        First_Flr_SF,
        Second_Flr_SF,
        Wood_Deck_SF,
        Open_Porch_SF
      ),
      .fns = mean
    ),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

Level 4: Use across with custom function

Similarly, we can use across() also inside of mutate(). But this time we also have to define a function that transforms square feet to square meters.

housing_dat |> 
  select(Neighborhood, contains('SF')) |> 
  mutate(
    across(
      .cols = c(
        BsmtFin_SF_1,
        BsmtFin_SF_2,
        Bsmt_Unf_SF,
        Total_Bsmt_SF,
        First_Flr_SF,
        Second_Flr_SF,
        Wood_Deck_SF,
        Open_Porch_SF
      ),
      .fns = \(sqft) sqft / 10.7639
    )
  ) |> 
  summarize(
    across(
      .cols = c(
        BsmtFin_SF_1,
        BsmtFin_SF_2,
        Bsmt_Unf_SF,
        Total_Bsmt_SF,
        First_Flr_SF,
        Second_Flr_SF,
        Wood_Deck_SF,
        Open_Porch_SF
      ),
      .fns = mean
    ),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

And of course you can use this technique to use custom function arguments as well.

housing_dat |> 
  select(Neighborhood, contains('SF')) |> 
  mutate(
    across(
      .cols = c(
        BsmtFin_SF_1,
        BsmtFin_SF_2,
        Bsmt_Unf_SF,
        Total_Bsmt_SF,
        First_Flr_SF,
        Second_Flr_SF,
        Wood_Deck_SF,
        Open_Porch_SF
      ),
      .fns = \(sqft) sqft / 10.7639
    )
  ) |> 
  summarize(
    across(
      .cols = c(
        BsmtFin_SF_1,
        BsmtFin_SF_2,
        Bsmt_Unf_SF,
        Total_Bsmt_SF,
        First_Flr_SF,
        Second_Flr_SF,
        Wood_Deck_SF,
        Open_Porch_SF
      ),
      .fns = \(sqft) mean(sqft, na.rm = TRUE)
    ),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

Level 5: Use tidyselect helpers in `across()`

Finally, we can also use the tidyselect helpers inside of acrouss(). This is incredibly convenient and makes the code much shorter.

housing_dat |> 
  select(Neighborhood, contains('SF')) |> 
  mutate(
    across(
      .cols = contains('SF'),
      .fns = \(sqft) sqft / 10.7639
    )
  ) |> 
  summarize(
    across(
      .cols = contains('SF'),
      .fns = mean
    ),
    .by = Neighborhood
  )
## # A tibble: 28 × 9
##    Neighborhood BsmtFin_SF_1 BsmtFin_SF_2 Bsmt_Unf_SF Total_Bsmt_SF First_Flr_SF
##                                                   
##  1 North_Ames          0.340       7.77          43.0          95.8        109. 
##  2 Gilbert             0.480       0.0636        58.6          81.3         87.4
##  3 Stone_Brook         0.341       2.76          70.8         145.         147. 
##  4 Northwest_A…        0.277       5.81          50.6         109.         122. 
##  5 Somerset            0.427       0.776         72.6         110.         110. 
##  6 Briardale           0.356       0.910         27.4          52.3         52.3
##  7 Northpark_V…        0.137       8.34          41.2          85.8         85.3
##  8 Northridge_…        0.371       0.847         80.8         151.         150. 
##  9 Bloomington…        0.392       0             91.0         121.         131. 
## 10 Northridge          0.335       8.40          56.4         133.         137. 
## # ℹ 18 more rows
## # ℹ 3 more variables: Second_Flr_SF , Wood_Deck_SF ,
## #   Open_Porch_SF

Bonus-Level: Use clean names

And to throw in a little bonus level for you, let me mention the clean_names() function from the {janitor} package. It transforms all column names into snake_case. That way, you’ll have much more programming friendly column names.

And I know: Sounds mundane but it will save you lots of time in the long run.

housing_dat |> 
  janitor::clean_names() |> 
  select(neighborhood, contains('_sf')) |> 
  mutate(
    across(
      .cols = contains('_sf'),
      .fns = \(sqft) sqft / 10.7639
    )
  ) |> 
  summarize(
    across(
      .cols = contains('_sf'),
      .fns = mean
    ),
    .by = neighborhood
  )
## # A tibble: 28 × 9
##    neighborhood        bsmt_fin_sf_1 bsmt_fin_sf_2 bsmt_unf_sf total_bsmt_sf
##                                                    
##  1 North_Ames                  0.340        7.77          43.0          95.8
##  2 Gilbert                     0.480        0.0636        58.6          81.3
##  3 Stone_Brook                 0.341        2.76          70.8         145. 
##  4 Northwest_Ames              0.277        5.81          50.6         109. 
##  5 Somerset                    0.427        0.776         72.6         110. 
##  6 Briardale                   0.356        0.910         27.4          52.3
##  7 Northpark_Villa             0.137        8.34          41.2          85.8
##  8 Northridge_Heights          0.371        0.847         80.8         151. 
##  9 Bloomington_Heights         0.392        0             91.0         121. 
## 10 Northridge                  0.335        8.40          56.4         133. 
## # ℹ 18 more rows
## # ℹ 4 more variables: first_flr_sf , second_flr_sf ,
## #   wood_deck_sf , open_porch_sf

Sending e-mails with {blastula}

Albert Rapp — Sat, 18 Jan 2025 23:00:00 GMT

E-mails are a surprisingly effective tool to

send out recurring reports/stats,
alert people that a long-running job finished,
or simply contact customers or other departments with user-specific information.

So in today’s video, I’m going to show you how the {blastula} package makes all of that possible for you. Let’s dive in.

Creating an email

The first thing you need to do is to compose an email. {blastula} gives you the compose_email() function for that. And for you convenience you can use Markdown notation using the md() function.

library(blastula)
compose_email(
  body = md(
  "Hi there 👋,
  
  This is an email to let you now that - **hooooray** 🥳 - your long running job **finished**.

  Best,

  Albert"
  )
)

If you run this, you’ll see that the HTML email message is displayed in RStudio’s viewer window.

Filling the email with data

Now as I said, e-mails could also contain some key data for stakeholders or customers. So, let’s create a fake dataset to simulate that:

dat_running <- tibble::tibble(
  job_name = 'My cool job',
  job_started = lubridate::now() - lubridate::minutes(57),
  job_ended = lubridate::now(),
  job_time = lubridate::interval(job_started, job_ended) /
    lubridate::minutes(1)
)
dat_running
## # A tibble: 1 × 4
##   job_name    job_started         job_ended           job_time
##                                         
## 1 My cool job 2025-01-18 19:53:19 2025-01-18 20:50:19     57.0

Then, we can assemble an HTML table, e.g. with {gt}.

library(gt)
## 
## Attaching package: 'gt'
## The following object is masked from 'package:blastula':
## 
##     md
tbl <- gt(dat_running) |> 
  fmt_datetime(columns = 2:3) |> 
  fmt_duration(
    columns = job_time, 
    input_units = 'minutes'
  )
tbl

job_name	job_started	job_ended	job_time
My cool job	2025-01-18 19:53:19	2025-01-18 20:50:19	57m

And then we can stick everything together with glue(). Just make sure to convert your {gt} table to raw html.

msg <- compose_email(
  body = md(
    glue::glue(
       "Hi there 👋,
  
      This is an email to let you now that - **hooooray** 🥳 - your long running job **finished**. 
      Here is some data on your long running job:
      
      {tbl |> gt::as_raw_html()}
    
      Best,

      Albert"
    )
  )
)
msg

Get email credentials

Nice. We have a semi-good-lookin’ email. Let’s send that to someone. For that, we have to create “smtp credentials” first.

There are multiple ways to do that. You can

create a credentials file with create_smtp_creds_file(),
store credentials in your computer’s key-value vault with create_smtp_creds_key(), or
retrieve your passwort using environment variables with creds_envvar().

They all work pretty much the same but here we will go for the one that uses environment variables. This one is the easiest to implement.

If you’re using gmail, outlook or office365 as your email provider, then it’s particularly simple. You’ll just have to provide creds_envvar() with

your user name (typically e-mail),
the name of the env-variable that stores your password, and
your provider.

In this case, I’m using gmail as a provider and I store my password in a variable called SMTP_PASSWORD.

my_email_creds <- creds_envvar(
  user = Sys.getenv('MY_GMAIL_ACCOUNT'),
  pass_envvar = 'SMTP_PASSWORD', 
  provider = 'gmail'
)

Here, I’m also using an env-variable for my e-mail but that’s purely optional. Also, it is worth noting that for gmails you cannot use your regular password. You will have to create a designated app password.

Sending out the email

Finally, sending out the email is pretty straightforward. Just take your output of compose_email() and pass it to smtp_send(). But don’t forget to include the credentials we’ve just created.

msg |> 
  smtp_send(
    from = Sys.getenv('MY_GMAIL_ACCOUNT'),
    to = "info@albert-rapp.de",
    subject = "Testing the `smtp_send()` function",
    credentials = my_email_creds
  )
## The email message was sent successfully.

Comparing pipes: Base-R |> vs {magrittr} %>%

Albert Rapp — Sat, 11 Jan 2025 23:00:00 GMT

Beginners are sometimes confused by the fact that

some R users use the native Base R pipe |> and
others use the {magrittr} pipe %>%.

So in today’s video, I want to compare the two and show you the strengths and weaknesses of each one. Let’s dive in.

Keyboard shortcut

Whatever pipe you use, you should definitely use the RStudio shortcut ctrl + shift + M. This is much quicker than writing it out. By default, this will throw the {magrittr} pipe. But you can change that in the settings.

Simple function chaining

The big advantage of the base-R pipe is that it can easily chain together a couple of functions whether any packages are loaded or not.

runif(100) |> round() |> mean()
## [1] 0.48

The same doesn’t work with the {magrittr} pipe because I have to load the package first.

runif(100) %>% round() %>%  mean()
## Error in runif(100) %>% round() %>% mean(): could not find function "%>%"

But if I do load something like the Tidyverse that contains {magrittr} it works fine.

library(tidyverse) 
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.5.1     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package () to force all conflicts to become errors
runif(100) %>% round() %>% mean()
## [1] 0.53

Form strictness

The nice thing about the {magrittr} pipe is that it isn’t as strict as the base-R pipe. For example, {magrittr} allows you to forget function calls and just use the function name.

runif(100) %>% round # works
runif(100) |> round  # Error function call with () is enforced
## Error: The pipe operator requires a function call as RHS (:2:15)

Standard scenario

I don’t think the strictness is much of a disadvantage, though. In most cases (at least in my 90% of pipe use cases), you’ll likely use the pipe with something like mutate() where you specify additional arguments anyway. In that scenario, both pipes work pretty much the same.

dat_with_super_long_name <- tibble(x = 1:3, y = 10:12)
dat_with_super_long_name |> 
  mutate(z = x + y)
## # A tibble: 3 × 3
##       x     y     z
##     
## 1     1    10    11
## 2     2    11    13
## 3     3    12    15
dat_with_super_long_name %>%
  mutate(z = x + y)
## # A tibble: 3 × 3
##       x     y     z
##     
## 1     1    10    11
## 2     2    11    13
## 3     3    12    15

Using a placeholder

Fans of the original {magrittr} pipe will tell you that it’s really cool to use the . operator as a placeholder. Rightfully so, this is a neat feature.

dat_with_super_long_name %>% lm(y ~ x, data = .)
## 
## Call:
## lm(formula = y ~ x, data = .)
## 
## Coefficients:
## (Intercept)            x  
##           9            1

Initially, the base-R pipe could not pull of such a stunt. However, since R 4.3.0. it has a placeholder too.

dat_with_super_long_name |> lm(y ~ x, data = _)
## 
## Call:
## lm(formula = y ~ x, data = dat_with_super_long_name)
## 
## Coefficients:
## (Intercept)            x  
##           9            1

Using multiple placeholders

At this point, fans of the . operator will shout “The dot operator is even cooler. It can be used multiple times!” And they are absolutely right about that. That’s pretty dope.

And for the unenlightened: By wrapping a subsequent function call into {}, you can use the . operator as many times as you’d like over there. In each instance, . will then represent the data that went into {}.

dat_with_super_long_name %>% {plot(.$x, .$y, cex = 3, lwd = 5)}

Sadly, the base pipe cannot do such a thing. Its strictness forbids {}.

## Error: { not allowed
dat_with_super_long_name |> {plot(_$x, _$y, cex = 3, lwd = 5)} 
## Error: function '{' not supported in RHS call of a pipe (:2:29)

A workaround for that would be to

define an anonymous function with \(.),
wrap that into parentheses, and then
call that function.

dat_with_super_long_name |> 
  (\(.) plot(.$x, .$y, cex = 3, lwd = 5))()

Shoutout to Isabella Velásquez’s blog post that taught me about this little trick.

Conditional flows

Now, sometimes people like to use if-statements in their pipe-chains. By combining the {magrittr} pipe with curly brackets and the . operator, this could look like this.

TRUE
FALSE

duplicate_flag <- TRUE
duplicates <- tibble(x = 1:3, z = 21:23)
dat_with_super_long_name %>%
  {
    if (duplicate_flag) {
      . |> left_join(duplicates, by = 'x')
    } else {
      .
    }
  } %>%
  summarize(across(everything(), mean))
## # A tibble: 1 × 3
##       x     y     z
##     
## 1     2    11    22

duplicate_flag <- FALSE
duplicates <- tibble(x = 1:3, z = 21:23)
dat_with_super_long_name %>%
  {
    if (duplicate_flag) {
      . |> left_join(duplicates, by = 'x')
    } else {
      .
    }
  } %>%
  summarize(across(everything(), mean))
## # A tibble: 1 × 2
##       x     y
##    
## 1     2    11

In the past, I have written code like this too. Nowadays, though, I try to break out such things into their own functions. Preferably, one with a descriptive function name.

That way,

the base-R pipe can handle this much better,
my original chain hopefully stays short, and
when I outsource the helper functions to a separate script, the function name hopefully still tells me what it does.

TRUE
FALSE

left_join_if_duplicate <- function(dat, duplicate_flag) {
  if (duplicate_flag) {
    dat |> left_join(duplicates, by = 'x') 
  } else {
    dat
  }
}
duplicate_flag <- TRUE
dat_with_super_long_name |> 
  left_join_if_duplicate(duplicate_flag) |> 
  summarize(across(everything(), mean))
## # A tibble: 1 × 3
##       x     y     z
##     
## 1     2    11    22

left_join_if_duplicate <- function(dat, duplicate_flag) {
  if (duplicate_flag) {
    dat |> left_join(duplicates, by = 'x') 
  } else {
    dat
  }
}
duplicate_flag <- FALSE
dat_with_super_long_name |> 
  left_join_if_duplicate(duplicate_flag) |> 
  summarize(across(everything(), mean))
## # A tibble: 1 × 2
##       x     y
##    
## 1     2    11

Getting started with SQL with R

Albert Rapp — Fri, 04 Oct 2024 22:00:00 GMT

In today’s blog post, I’m showing you how to work with databases from within R. As always, you can find the video version of this blog post on YouTube:

Establish a connection

Let us start by setting up a database connection.

library(odbc)
library(DBI)

con <- dbConnect(
  drv = odbc(),
  driver = "SQLite3 Driver",
  database = ':memory:',
  server = 'localhost',
  uid = 'user_name',
  pwd = 'my_password',
  port = 5432
)
## Error in `dbConnect()`:
## ! ODBC failed with error 00000 from [unixODBC][Driver Manager].
## ✖ Can't open lib 'SQLite3 Driver' : file not found
## ℹ From 'nanodbc/nanodbc.cpp:1150'.

If there is an error, find out if you have the ODBC driver installed on your computer. You can check that with

odbcListDrivers()
##                  name   attribute                                    value
## 1              SQLite Description                       SQLite ODBC Driver
## 2              SQLite      Driver                         libsqliteodbc.so
## 3              SQLite       Setup                         libsqliteodbc.so
## 4              SQLite  UsageCount                                        1
## 5             SQLite3 Description                      SQLite3 ODBC Driver
## 6             SQLite3      Driver                        libsqlite3odbc.so
## 7             SQLite3       Setup                        libsqlite3odbc.so
## 8             SQLite3  UsageCount                                        1
## 9     PostgreSQL ANSI Description    PostgreSQL ODBC driver (ANSI version)
## 10    PostgreSQL ANSI      Driver                             psqlodbca.so
## 11    PostgreSQL ANSI       Setup                          libodbcpsqlS.so
## 12    PostgreSQL ANSI       Debug                                        0
## 13    PostgreSQL ANSI     CommLog                                        1
## 14    PostgreSQL ANSI  UsageCount                                        1
## 15 PostgreSQL Unicode Description PostgreSQL ODBC driver (Unicode version)
## 16 PostgreSQL Unicode      Driver                             psqlodbcw.so
## 17 PostgreSQL Unicode       Setup                          libodbcpsqlS.so
## 18 PostgreSQL Unicode       Debug                                        0
## 19 PostgreSQL Unicode     CommLog                                        1
## 20 PostgreSQL Unicode  UsageCount                                        1

Aha! The driver name seems to be different. Using the correct driver name, the previous code works nicely.

con <- dbConnect(
  drv = odbc(),
  driver = "SQLite3",
  database = ':memory:',
  server = 'localhost',
  uid = 'user_name',
  pwd = 'my_password',
  port = 5432
)
con
##  :memory:
##   Database: :memory:
##   SQLite Version: 3.36.0

Cool, we have a connection but what if there wasn’t a SQLite driver installed on my system? In that case, I could install it the driver by first checking what the name for the installation is. For Ubuntu users that would be

apt-cache search sqlite odbc
## fp-units-db - Free Pascal - database-library units dependency package
## fp-units-db-3.2.2 - Free Pascal - database-library units
## libcppdb-dev - SQL Connectivity Library (development files)
## libcppdb-mysql0 - SQL Connectivity Library (MySQL backend)
## libcppdb-odbc0 - SQL Connectivity Library (odbc backend)
## libcppdb-postgresql0 - SQL Connectivity Library (PostgreSQL backend)
## libcppdb-sqlite3-0 - SQL Connectivity Library (sqlite3 backend)
## libcppdb0 - SQL Connectivity Library (core library)
## libghc-hdbc-sqlite3-dev - Sqlite v3 HDBC (Haskell Database Connectivity) Driver for GHC
## libghc-hdbc-sqlite3-doc - Sqlite v3 HDBC (Haskell Database Connectivity) Documentation
## libghc-hdbc-sqlite3-prof - Sqlite v3 HDBC Driver for GHC; profiling libraries
## libpoco-doc - Documentation for POCO - The C++ Portable Components
## libreoffice-base-drivers - Database connectivity drivers for LibreOffice
## libsqliteodbc - ODBC driver for SQLite embedded database
## php-db - Database Abstraction Layer
## ruby-sequel - Simple, flexible, and powerful SQL database access toolkit for Ruby

And from there I could identify that libsqliteodbc is the thing I need to install. On Ubuntu, that can be done with

sudo apt-get install libsqliteodbc

Anyway, enough with the technical stuff.

Use database tables

Let’s check what tables we have in our database right now.

con |> 
  dbListTables()
## character(0)

Well, that’s boring. No tables yet. Let’s create one. Let’s use the penguins from the {palmerpenguins} package and throw that into a table called palmerpenguins.

con |> 
  dbCreateTable(
    name = 'palmerpenguins',
    fields = palmerpenguins::penguins
  )

Now, we can see that we have a table in our database.

con |> 
  dbListTables()
## [1] "palmerpenguins"

And we could fetch the data as well.

con |> 
  dbReadTable('palmerpenguins') |> 
  tibble::as_tibble() # nicer output with tibbles
## # A tibble: 0 × 8
## # ℹ 8 variables: species , island , bill_length_mm ,
## #   bill_depth_mm , flipper_length_mm , body_mass_g , sex ,
## #   year

Looks like the table is still empty and only the columns (and their type) were initialized. Time to append the actual data.

con |> 
  dbAppendTable(
    name = 'palmerpenguins',
    value = palmerpenguins::penguins
  )

Once that is executed, we should now see data inside of our table.

penguins_db <- con |> 
  dbReadTable('palmerpenguins') |> 
  tibble::as_tibble()
penguins_db
## # A tibble: 344 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 334 more rows
## # ℹ 2 more variables: sex , year

Excellent. We have our data right here. And once we have the full data locally, we can operate on it like we normally would, e.g.

penguins_db |> 
  dplyr::filter(!is.na(sex), bill_length_mm > 50) |> 
  dplyr::select(bill_length_mm, sex)
## # A tibble: 52 × 2
##    bill_length_mm sex  
##              
##  1           50.2 male 
##  2           59.6 male 
##  3           50.5 male 
##  4           50.5 male 
##  5           50.1 male 
##  6           50.4 male 
##  7           54.3 male 
##  8           50.7 male 
##  9           51.1 male 
## 10           52.5 male 
## # ℹ 42 more rows

Get data with SQL

But what if we wanted to have only have the data return already the filtered data? That way, we don’t have to possibly download loads and loads of data from the database only to filter for and select specific data locally.

Well, we could just use a simple SQL statement to get the data that we want. The key function to make that possible is the dbGetQuery() function.

con |> 
  dbGetQuery(
    'SELECT bill_length_mm, sex 
    FROM palmerpenguins
    WHERE bill_length_mm > 50 and sex IS NOT NULL;'
  )  |> 
  tibble::as_tibble()
## # A tibble: 52 × 2
##    bill_length_mm sex  
##              
##  1           50.2 male 
##  2           59.6 male 
##  3           50.5 male 
##  4           50.5 male 
##  5           50.1 male 
##  6           50.4 male 
##  7           54.3 male 
##  8           50.7 male 
##  9           51.1 male 
## 10           52.5 male 
## # ℹ 42 more rows

dtplyr

Another possibility is to use the {dbplyr} package. This package lets you use the syntax from the {dplyr} package and translates that automatically to SQL. You just have to know three particular things to use that:

You need to tell {dbplyr} which table you want to use on the database side with dplyr::tbl().
You can have a look at the SQL code with dplyr::show_query().
If you want to have the full query results locally (and not just a preview), then you need to collect the data using dplyr::collect().

tbl_palmerpenguins <- dplyr::tbl(con, 'palmerpenguins')
extraction <- tbl_palmerpenguins |> 
  dplyr::filter(!is.na(sex), bill_length_mm > 50) |> 
  dplyr::select(bill_length_mm, sex)
extraction |> dplyr::show_query()
## 
## SELECT "bill_length_mm", "sex"
## FROM "palmerpenguins"
## WHERE (NOT(("sex" IS NULL))) AND ("bill_length_mm" > 50.0)
extraction |> dplyr::collect()
## # A tibble: 52 × 2
##    bill_length_mm sex  
##              
##  1           50.2 male 
##  2           59.6 male 
##  3           50.5 male 
##  4           50.5 male 
##  5           50.1 male 
##  6           50.4 male 
##  7           54.3 male 
##  8           50.7 male 
##  9           51.1 male 
## 10           52.5 male 
## # ℹ 42 more rows

This also works with mutate() calls that change the returned data on the database side befor downloading it.

tbl_palmerpenguins |> 
  dplyr::filter(!is.na(sex), bill_length_mm > 50) |> 
  dplyr::select(bill_length_mm, sex) |> 
  dplyr::mutate(
    sex = if_else(sex == 'male', 'MALE', 'FEMALE')
  ) |> 
  dplyr::show_query()
## 
## SELECT
##   "bill_length_mm",
##   CASE WHEN ("sex" = 'male') THEN 'MALE' WHEN NOT ("sex" = 'male') THEN 'FEMALE' END AS "sex"
## FROM "palmerpenguins"
## WHERE (NOT(("sex" IS NULL))) AND ("bill_length_mm" > 50.0)

So in this case you would get

tbl_palmerpenguins |> 
  dplyr::filter(!is.na(sex), bill_length_mm > 50) |> 
  dplyr::select(bill_length_mm, sex) |> 
  dplyr::mutate(
    sex = if_else(sex == 'male', 'MALE', 'FEMALE')
  ) |> 
  dplyr::collect()
## # A tibble: 52 × 2
##    bill_length_mm sex  
##              
##  1           50.2 MALE 
##  2           59.6 MALE 
##  3           50.5 MALE 
##  4           50.5 MALE 
##  5           50.1 MALE 
##  6           50.4 MALE 
##  7           54.3 MALE 
##  8           50.7 MALE 
##  9           51.1 MALE 
## 10           52.5 MALE 
## # ℹ 42 more rows

But it is important to note that this does not actually update the underlying data. You see, if you connect the data once more (without the mutate() call), then you get the original data:

tbl_palmerpenguins |> 
  dplyr::filter(!is.na(sex), bill_length_mm > 50) |> 
  dplyr::select(bill_length_mm, sex) |> 
  dplyr::collect()
## # A tibble: 52 × 2
##    bill_length_mm sex  
##              
##  1           50.2 male 
##  2           59.6 male 
##  3           50.5 male 
##  4           50.5 male 
##  5           50.1 male 
##  6           50.4 male 
##  7           54.3 male 
##  8           50.7 male 
##  9           51.1 male 
## 10           52.5 male 
## # ℹ 42 more rows

Programatically assembling SQL queries with `{glue}`

Another way to assemble SQL queries is using the glue_sql() function from the {glue} package. This can be particularly useful when you are already familiar with SQL and don’t want to learn the {dtplyr} syntax.

For example, if you have vectors that describes particular columns and values your interested in. For our penguins, this could be something like this:

selected_tbl <- 'palmerpenguins'
selected_cols <- c('bill_length_mm', 'sex')
selected_species <- c('Adelie', 'Chinstrap')

Then with the glue_sql() command you can easily throw these variables into a SQL-query that you write yourself.

glue::glue_sql(
  'SELECT {selected_cols}
  FROM {selected_tbl}
  WHERE species IN ({selected_species})',
  .con = con
)
##  SELECT 'bill_length_mm'
## FROM 'palmerpenguins'
## WHERE species IN ('Adelie')
##  SELECT 'sex'
## FROM 'palmerpenguins'
## WHERE species IN ('Chinstrap')

Well, this doesn’t look right. It’s two SQL queries instead of one. But the nice thing about glue_sql() is that you can collapse vectors with the * character.

glue::glue_sql(
  'SELECT {selected_cols*}
  FROM {selected_tbl}
  WHERE species IN ({selected_species*})',
  .con = con
)
##  SELECT 'bill_length_mm', 'sex'
## FROM 'palmerpenguins'
## WHERE species IN ('Adelie', 'Chinstrap')

But if you know SQL, you know that this code is not quite right. Check out what happens if you run this query.

incorrect_query <- glue::glue_sql(
  'SELECT {selected_cols*}
  FROM {selected_tbl}
  WHERE species IN ({selected_species*})',
  .con = con
)
con |> 
  dbGetQuery(incorrect_query) |> 
  tibble::as_tibble()
## # A tibble: 220 × 2
##    `'bill_length_mm'` `'sex'`
##                    
##  1 bill_length_mm     sex    
##  2 bill_length_mm     sex    
##  3 bill_length_mm     sex    
##  4 bill_length_mm     sex    
##  5 bill_length_mm     sex    
##  6 bill_length_mm     sex    
##  7 bill_length_mm     sex    
##  8 bill_length_mm     sex    
##  9 bill_length_mm     sex    
## 10 bill_length_mm     sex    
## # ℹ 210 more rows

That’s not quite right, is it? Well, the thing is: In our SQL-query the quotation marks are not correct. Have a look:

incorrect_query
##  SELECT 'bill_length_mm', 'sex'
## FROM 'palmerpenguins'
## WHERE species IN ('Adelie', 'Chinstrap')

For the things in the SELECT and FROM part we shouldn’t use single quotes '. In SQLite, these are reserved for literal strings like 'Adelie' and 'Chinstrap'. If you want to use identifiers like column and tables names, then you either don’t use any quotation marks are double quotation marks ". You can tell glue_sql() about the difference by wrapping identifiers into backticks `.

correct_query <- glue::glue_sql(
  'SELECT {`selected_cols`*}
  FROM {`selected_tbl`}
  WHERE species IN ({selected_species*})',
  .con = con
)
correct_query
##  SELECT "bill_length_mm", "sex"
## FROM "palmerpenguins"
## WHERE species IN ('Adelie', 'Chinstrap')

So, with this query, you can now run the code from before and get the right results.

con |> 
  dbGetQuery(correct_query) |> 
  tibble::as_tibble()
## # A tibble: 220 × 2
##    bill_length_mm sex   
##               
##  1           39.1 male  
##  2           39.5 female
##  3           40.3 female
##  4           NA     
##  5           36.7 female
##  6           39.3 male  
##  7           38.9 female
##  8           39.2 male  
##  9           34.1   
## 10           42     
## # ℹ 210 more rows

Transactions

Now, let me teach you one more thing that’s essential to working with databases. Namely, transactions. You see, you can often want to have changes to your database run transactionally.

This means that if you update multiple rows, you want to be sure that either all rows were updated successfully or no rows were changed. That’s what transactions do. Here’s how they work:

First, you begin a transaction:

con |> dbBegin()

Then, you make changes to the database like appending more rows to a table.

con |> 
  dbAppendTable(
    'palmerpenguins',
    palmerpenguins::penguins
  )

And now you could see your changes:

con |> 
  dbReadTable('palmerpenguins') |> 
  tibble::as_tibble()
## # A tibble: 688 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 678 more rows
## # ℹ 2 more variables: sex , year

See how you have double the amount of rows now? This means you have successfully appended more rows to the database table. But not so fast. You can revert all your changes since starting your transaction with dbBegin() by calling dbRollback().

con |> dbRollback()

And now if you grab the table again, you can see that you’re back where you started.

con |> 
  dbReadTable('palmerpenguins') |> 
  tibble::as_tibble()
## # A tibble: 344 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 334 more rows
## # ℹ 2 more variables: sex , year

Of course, there’s also a way to make changes permanent. And that happens with dbCommit().

# Start transaction
con |> dbBegin()
# Make changes
con |> 
  dbAppendTable(
    'palmerpenguins',
    palmerpenguins::penguins
  )
# Commit to changes
con |> dbCommit()
# Changes are permanent now.
con |> 
  dbReadTable('palmerpenguins') |> 
  tibble::as_tibble()
## # A tibble: 688 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 678 more rows
## # ℹ 2 more variables: sex , year

You might be wondering how this could be useful. So let me show you how.

Imagine that we have a constraint on the species column inside of our data set. We could make that happen by creating a new table palmerpenguins_new with a bit of SQL.

allowed_species <- c('Adelie', 'Chinstrap', 'Gentoo')
query <- glue::glue_sql(
  "CREATE TABLE palmerpenguins_new (
    species TEXT NOT NULL CHECK(species IN ({allowed_species*})),
    island TEXT,
    bill_length_mm REAL,
    bill_depth_mm REAL,
    flipper_length_mm REAL,
    body_mass_g REAL,
    sex TEXT,
    year INTEGER
  );",
  .con = con
)
con |> dbExecute(query)
## [1] 0

Clearly, we could append the original palmerpenguins data set to this table (as it fulfills the constraint).

con |> 
  dbAppendTable('palmerpenguins_new', value = palmerpenguins::penguins)
con |> 
  dbReadTable('palmerpenguins_new') |> 
  tibble::as_tibble()
## # A tibble: 344 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 334 more rows
## # ℹ 2 more variables: sex , year

But what about a data set that has a row in it with a wrong species. Like this one here:

fake_dat <- dplyr::bind_rows(
  palmerpenguins::penguins[1,],
  tibble::tibble(species = 'NEW SPECIES')
)
fake_dat
## # A tibble: 2 × 8
##   species     island  bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                   
## 1 Adelie      Torger…           39.1          18.7               181        3750
## 2 NEW SPECIES               NA            NA                  NA          NA
## # ℹ 2 more variables: sex , year

Well, appending that should work for the first row but not for the second one. Let’s see what happens if we try anyway:

# Will work
con |> 
  dbAppendTable(
    'palmerpenguins_new', 
    value = fake_dat[1, ]
  )
# Will fail
con |> 
  dbAppendTable(
    'palmerpenguins_new', 
    value = fake_dat[2, ]
  )
## Error in eval(expr, envir, enclos): nanodbc/nanodbc.cpp:1783: 00000
## [SQLite]CHECK constraint failed: species IN ('Adelie', 'Chinstrap', 'Gentoo') (19)

Nice. We get an error because the constrains were not fulfilled for the second row. As we should. But because we the first operation was successful, the first (valid) row was still appended. Notice how there are 345 rows in the database table now as opposed to 344 rows from the original data set.

con |> 
  dbReadTable('palmerpenguins_new') |> 
  tibble::as_tibble()
## # A tibble: 345 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 335 more rows
## # ℹ 2 more variables: sex , year

That’s unfortunate. We now have to figure out what operations worked and which didn’t. In real-world settings where we often try to string together multiple operations, this can become tediuous quite fast. That’s why it’s common to enforce that either all opeartions work or everything is reverted to normal.

And that’s where transactions come in. Together with tryCatch() they are an invaluable tool to ensure database consistency.

con |> dbBegin()
tryCatch({
  # Will work
  con |> 
    dbAppendTable(
      'palmerpenguins_new', 
      value = fake_dat[1, ]
    )
  # Will fail
  con |> 
    dbAppendTable(
      'palmerpenguins_new', 
      value = fake_dat[2, ]
    )
  con |> dbCommit() # Commit to changes if no error until here
  cli::cli_alert_success("Data uploaded successfully")
}, error = function(e) {
  con |> dbRollback() # Rollback if there was an error
  cli::cli_abort("Data couln't be uploaded successfully. There was the following error {e$message}")
})
## Error in `value[[3L]]()`:
## ! Data couln't be uploaded successfully. There was the following error
##   nanodbc/nanodbc.cpp:1783: 00000 [SQLite]CHECK constraint failed: species IN
##   ('Adelie', 'Chinstrap', 'Gentoo') (19)

Now you’ll see that we are still left with the 345 rows instead of adding yet another row (even though the first operation worked.)

con |> 
  dbReadTable('palmerpenguins_new') |> 
  tibble::as_tibble()
## # A tibble: 345 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 335 more rows
## # ℹ 2 more variables: sex , year

And with that we have concluded our little trip into databases with R. Of course there’s lots more to learn about SQL and database interactions from within R. But for now, I think this is a good place to stop.

Creating fancy interactive tables using Internet data with rvest and reactable

Albert Rapp — Sat, 31 Aug 2024 22:00:00 GMT

This blog post will be a long one as we’ll create this fancy interactive table:

Code

Creating interactive tables with reactable

Albert Rapp — Sat, 17 Aug 2024 22:00:00 GMT

The R programming language has a rich ecosystem of packages that are fantastic for creating beautiful production-grade tables from within R. Today, I’m showing you that one package that makes it really easy (mostly) to create interactive tables. Namely, I’m going to show you {reactable}. 🥳

If you want to see a video version of this blog post, you can find it on YouTube:

Fake data

Let’s first create a dummy data set using one of {gt}’s built-in data sets. {gt} has a lot of those so we might as well use it even if we don’t use {gt} to create the table in the end.

library(tidyverse)
library(reactable)
hawaiian_sales <- gt::pizzaplace |> 
  filter(name == 'hawaiian') |> 
  mutate(
    month = month(
      date, label = TRUE, abbr = FALSE,
      locale = 'en_US.UTF-8' # English month names
    ),
    quarter = paste0('Q', quarter(date))
  ) |> 
  summarise(
    sales = n(),
    revenue = sum(price),
    .by = c(month, quarter)
  )
hawaiian_sales
## # A tibble: 12 × 4
##    month     quarter sales revenue
##               
##  1 January   Q1        185   2443.
##  2 February  Q1        198   2633 
##  3 March     Q1        217   2878.
##  4 April     Q2        219   2868.
##  5 May       Q2        198   2688 
##  6 June      Q2        189   2564.
##  7 July      Q3        195   2620.
##  8 August    Q3        201   2679.
##  9 September Q3        196   2616.
## 10 October   Q4        188   2515.
## 11 November  Q4        227   2953.
## 12 December  Q4        209   2817.

Base Layer

To create a table all we have to to is to pass the data to the reactable function.

reactable(hawaiian_sales)

The nice thing is that this is interactive out of the box. By clicking onto the column names, you can sort the rows.

Use better column names

Unlike {gt}, the {reactable} package doesn’t allow to change the table step by step by chaining pipes. Instead, you will have to use one of the many arguments of reactable() and helper functions to get things done. For example, to set nicer column names you can use the columns argument with a list of column definitions (using the colDef() helper)

reactable(
  hawaiian_sales,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(name = 'Month'),
    sales = colDef(name = 'Sales'),
    revenue = colDef(name = 'Revenue')
  )
)

Title & Subtitle

For adding a nice title and subtitle to your plot, you can either use some custom HTML & CSS tricks or you just use the {reactablefmtr} package.

reactable(
  hawaiian_sales,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(name = 'Month'),
    sales = colDef(name = 'Sales'),
    revenue = colDef(name = 'Revenue')
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Format numbers

The numbers in the revenue column correspond to dollar amounts. We can format them by specifying a column format inside of colDef() with help from the colFormat() helper function.

reactable(
  hawaiian_sales,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(name = 'Month'),
    sales = colDef(name = 'Sales'),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE)
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Add groups

Now, I want to structure my tables into quarters. The easiest way to do that is to use the quarter column in our data set for grouping. The cool thing about {reactable} is that it’s really easy and the output becomes nicely interactive out of the box. All you have to do is set the groupBy argument.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(name = 'Month'),
    sales = colDef(name = 'Sales'),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE)
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Add summaries

You can add group summaries by using the aggregate argument inside of colDef() and setting it to one of the built-in aggregate functions like "mean" or "sum". If you want to do something custom, you can do that, but then you will have to write a custom JavaScript function for that.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(name = 'Month'),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum'
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum'
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Make table searchable

If we wanted to make our table more interactive, we could make the month column filterable. That way, we can look for particular columns. In that case, it probably makes sense to have the groups unfolded by default.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE, # Expand rows by default
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE  # Make column filterable
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum'
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum'
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Add a total footer

There are two ways to add a total footer to a column. One, you can use an R function that takes the values and column name as an argument and calculates the result from that. Or two, you can use a JavaScript function to the same thing but dynamically. Let’s first start with the R-approach and then see its drawback.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE, 
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE  
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum',
      footer = function(values, name) {
        sum(values)
      }
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum',
      footer = function(values, name) {
        sum(values) |> scales::dollar()
      }
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

But if you filter for January now, you will see that the overall total doesn’t update. That’s not something that R can do for you. That’s where JavaScript comes in. Thankfully, the reactable cookbook gives you the JS code you need.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE 
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total
      }"),
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total.toLocaleString('en-US', { style: 'currency', currency: 'USD' })
      }")
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Change row styling

Finally, to add a little bit more style and visual structure let us make the group rows blue. Let’s first try to change the theme() argument with the reactableTheme() helper function.

With that we can inject a bit of CSS to our table. The {htmltools} package makes it easy to combine multiple style instructions.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE 
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total
      }"),
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total.toLocaleString('en-US', { style: 'currency', currency: 'USD' })
      }")
    )
  ),
  theme = reactableTheme(
    rowGroupStyle = htmltools::css(
      background =  '#E7EDF3', 
      borderLeft = '2px solid #104E8B' 
    )
  )
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Unfortunately, this didn’t do what we want. This seems to target all cells in our table because they are all a row of some group. But to only highlight the header row of each group we will have to proceed differently.

For that, we need to write a JavaScript function that takes the rowInfo object as an argument and returns a JSON object with camelCased style properties. Thankfully, the reactable cookbook shows you exactly what you need.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE 
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total
      }"),
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total.toLocaleString('en-US', { style: 'currency', currency: 'USD' })
      }")
    )
  ),
  rowStyle = JS(
    "function(rowInfo) {
      if (rowInfo.level == 0) { // corresponds to row group
        return { 
          background: '#E7EDF3', 
          borderLeft: '2px solid #104E8B',
          fontWeight: 600
        }
      } 
    }"
  ),
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Change footer style

But defining plain CSS without having to wrap it in JS code can still be useful. For example, you could make the footer a bit nicer.

reactable(
  hawaiian_sales,
  groupBy = 'quarter',
  defaultExpanded = TRUE,
  columns = list(
    quarter = colDef(name = 'Quarter'),
    month = colDef(
      name = 'Month',
      filterable = TRUE 
    ),
    sales = colDef(
      name = 'Sales',
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total
      }"),
      footerStyle = htmltools::css(
        font_weight = 600,
        border_top = '2px solid black'
      )
    ),
    revenue = colDef(
      name = 'Revenue',
      format = colFormat(currency = 'USD', separators = TRUE),
      aggregate = 'sum',
      footer =  JS("function(column, state) {
        let total = 0
        state.sortedData.forEach(function(row) {
          total += row[column.id]
        })
        return total.toLocaleString('en-US', { style: 'currency', currency: 'USD' })
      }"),
      footerStyle = htmltools::css(
        font_weight = 600,
        border_top = '2px solid black'
      )
    )
  ),
  rowStyle = JS(
    "function(rowInfo) {
      if (rowInfo.level == 0) { // corresponds to row group
        return { 
          background: '#E7EDF3', 
          borderLeft: '2px solid #104E8B',
          fontWeight: 600
        }
      } 
    }"
  ),
) |> 
  reactablefmtr::add_title(
    title = 'Hawaiian Pizza Sales in 2015'
  ) |> 
  reactablefmtr::add_subtitle(
    subtitle = 'Based on the fake pizzaplace data from `{gt}`',
    font_weight = 'normal'
  )

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

An R User’s Guide to Other Programming Languages

Albert Rapp — Mon, 12 Aug 2024 22:00:00 GMT

Real-world problem often happen at the intersection of many areas. For example, maybe you want to build a web app for easier data ingestion. As an R user that’s no problem. You can easily dabble into the world of web development using the fantastic Shiny package.

But if you’re in that world, you will soon reach the boundaries of where using only R code can get you out of trouble. You will soon find that any meaningful Shiny app will have to sooner rather than later deal with problems that happen at the intersection of R and traditional webdev languages like HTML, CSS & JS. That’s where it helps to know a bit about these languages.

Similarly, you may want to incorporate a cool machine learning model into your app that’s only available in Python. In that case, knowing how to handle Python can be beneficial. Even if you do most of your scripting in R.

Or if some part of your R analysis is slow, it can be helpful to refactor some of your code in a compiled programming language like C++. That way, some of your bottle-neck functions can become blazingly fast all of a sudden.

And there are lots more examples where knowing your way around other programming languages can be really helpful. No one is saying that you need to give up R and become an expert in the other languages, though. You can make great progress by just knowing the basics of a language and combining that with your programming knowledge that you have already acquired through R programming.

In that spirit, this collection of resources is supposed to help you ease your way into other programming languages. The goal here is to highlight resources that will give you a gentle nudge towards these other languages when coming from R. Some of the resources I have created, some of them are things I’ve collected from the wider R community. If I forgot one of your favorite resources, don’t hesitate to put them into the comments.

Python

Probably the most popular data science programming language currently is Python. It’s particularly strong in the ML field.

Emily Riederer put together a fantastic series on the “Rgonomics” of Python:
- The first installment is a great way to learn about Python generally
- The second installment on polars is a great primer on polars which is a great framework for data wrangling.
- The third installment is a more general primer on Pythonic concepts like list comprehension.
Christoph Scheuch’s TidyIntelligence Blog put together a couple of great comparison blog posts:
- For data visualization: ggplot2 vs matplotlib and ggplot2 vs plotnins
- For data wrangling: dplyr vs pandas, dplyr vs polars, dplyr vs siuba and dplyr vs ibis

HTML & CSS

If you ever want to make your {gt}/{flextable}/{reactable} tables or your Shiny app or your Quarto documents look nice, then there’s no way around these two languages. A lot of the things that these languages can offer you, can be practiced from within R using the {htmltools} package.

I have created a “Web Dev for R Users” video series, that will get you started:
Other than that, I found the Outstanding User Interfaces with Shiny book by David Granjon fantastic.

Javascript

Javascript (JS) is another language that is used everywhere within web development. If HTML & CSS make your documents, tables and apps look nice, JS makes them interactive.

A nice way to dip your toes into the JS waters comes via Observable plots. They give you a nice way to to create charts using the grammar of graphics (just like {ggplot2}). A nice side-by-side comparison can be found in this fantastic guide by Allison Horst
There is also a nice video series for Observable plots.
A nice and free resource is also the JavaScript for R book by John Coene
If you want to take a deeper dive into web development with Javascript, you may also enjoy the free “Deep Dive Into Modern Web Development” course (even if it’s not R centered)

Calendar Plots With ggplot2

Albert Rapp — Sat, 03 Aug 2024 22:00:00 GMT

Take a look at this chart:

Clearly, it shows a periodic behavior. At some point of the week there are always much less flights scheduled than on other days of the week. This begs the question: “On what day are there less flights?”

It would be great if there’s a visualization that can show that more clearly. Thankfully, there is. Namely, there are calendar plots that can do just that. And with {ggplot2} it’s not that hard to create them.

In this blog post I show you how. All of the code can be found here. For more detailed explanations, check out the corresponding YT video:

Load the tidyverse and take a look at our data

library(tidyverse)
flights <- nycflights13::flights
flights
## # A tibble: 336,776 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##                                       
##  1  2013     1     1      517            515         2      830            819
##  2  2013     1     1      533            529         4      850            830
##  3  2013     1     1      542            540         2      923            850
##  4  2013     1     1      544            545        -1     1004           1022
##  5  2013     1     1      554            600        -6      812            837
##  6  2013     1     1      554            558        -4      740            728
##  7  2013     1     1      555            600        -5      913            854
##  8  2013     1     1      557            600        -3      709            723
##  9  2013     1     1      557            600        -3      838            846
## 10  2013     1     1      558            600        -2      753            745
## # ℹ 336,766 more rows
## # ℹ 11 more variables: arr_delay , carrier , flight ,
## #   tailnum , origin , dest , air_time , distance ,
## #   hour , minute , time_hour

Count flights per date

From the first three columns we can easily create a date and then count how often each date appears

date_counts <- flights |> 
  mutate(
    date = make_date(
      year = year, month = month, day = day
    )
  ) |> 
  count(date) 
date_counts
## # A tibble: 365 × 2
##    date           n
##         
##  1 2013-01-01   842
##  2 2013-01-02   943
##  3 2013-01-03   914
##  4 2013-01-04   915
##  5 2013-01-05   720
##  6 2013-01-06   832
##  7 2013-01-07   933
##  8 2013-01-08   899
##  9 2013-01-09   902
## 10 2013-01-10   932
## # ℹ 355 more rows

Get days of the month, week day, week of month

date_counts_w_labels <- date_counts |> 
  mutate(
    day = mday(date),
    month = month(
      date, label = T, abbr = F, locale = 'en_US.UTF-8'
    ),
    wday = wday(date, label = T, locale = 'en_US.UTF-8'),
    week = stringi::stri_datetime_fields(date)$WeekOfMonth
  )
date_counts_w_labels
## # A tibble: 365 × 6
##    date           n   day month   wday   week
##               
##  1 2013-01-01   842     1 January Tue       1
##  2 2013-01-02   943     2 January Wed       1
##  3 2013-01-03   914     3 January Thu       1
##  4 2013-01-04   915     4 January Fri       1
##  5 2013-01-05   720     5 January Sat       1
##  6 2013-01-06   832     6 January Sun       2
##  7 2013-01-07   933     7 January Mon       2
##  8 2013-01-08   899     8 January Tue       2
##  9 2013-01-09   902     9 January Wed       2
## 10 2013-01-10   932    10 January Thu       2
## # ℹ 355 more rows

Create first faceted plot

labels_color <- 'grey30'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3)

Make shapes square-ish

labels_color <- 'grey30'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE)

Make nicer color

labels_color <- 'grey30'
schedueled_color <- '#009E73'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  )

Remove grid lines

labels_color <- 'grey30'
schedueled_color <- '#009E73'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void()

Add titles, caption, and subtitle

labels_color <- 'grey30'
schedueled_color <- '#009E73'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  )

Move legend to top

labels_color <- 'grey30'
schedueled_color <- '#009E73'

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  ) +
  theme(
    legend.position = 'top'
  )

Style legend

labels_color <- 'grey30'
schedueled_color <- '#009E73'
bar_width_cm <- 15
bar_height_cm <- 0.3

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  ) +
  theme(
    legend.position = 'top'
  ) +
  guides(
    fill = guide_colorbar(
      barwidth = unit(bar_width_cm, 'cm'),
      barheight = unit(bar_height_cm, 'cm'),
      title.position = 'top',
      title.hjust = 0,
      title.vjust = 0,
      frame.colour = labels_color
    )
  )

Style texts & spacing

labels_color <- 'grey30'
schedueled_color <- '#009E73'
bar_width_cm <- 15
bar_height_cm <- 0.3
font_family <- 'Fira Sans'
bar_labels_size <- 11
month_size <- 12

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  ) +
  theme(
    legend.position = 'top',
    text = element_text(
      color = labels_color, 
      family = font_family
    ),
    plot.title = element_text(
      size = 24, 
      margin = margin(t = 0.25, b = 0.25, unit = 'cm')
    ),
    plot.subtitle = element_text(
      size = 16, 
      margin = margin(b = 0.5, unit = 'cm')
    ),
    plot.caption = element_text(
      size = 10, 
      margin = margin(b = 0.25, unit = 'cm')
    ),
    legend.text = element_text(size = bar_labels_size),
    legend.title = element_text(size = 14),
    strip.text = element_text(
      hjust = 0, 
      size = month_size,
      margin = margin(b = 0.25, unit = 'cm')
    )
  ) +
  guides(
    fill = guide_colorbar(
      barwidth = unit(bar_width_cm, 'cm'),
      barheight = unit(bar_height_cm, 'cm'),
      title.position = 'top',
      title.hjust = 0,
      title.vjust = 0,
      frame.colour = labels_color
    )
  )

Add text labels into the boxes

labels_color <- 'grey30'
schedueled_color <- '#009E73'
bar_width_cm <- 15
bar_height_cm <- 0.3
font_family <- 'Fira Sans'
bar_labels_size <- 11
month_size <- 12
nudge_labels <- 0.25
labels_size <- 3

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  geom_text(
    aes(label = day),
    nudge_x = nudge_labels,
    nudge_y = nudge_labels,
    col = labels_color,
    size = labels_size,
    family = font_family
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  ) +
  theme(
    legend.position = 'top',
    text = element_text(
      color = labels_color, 
      family = font_family
    ),
    plot.title = element_text(
      size = 24, 
      margin = margin(t = 0.25, b = 0.25, unit = 'cm')
    ),
    plot.subtitle = element_text(
      size = 16, 
      margin = margin(b = 0.5, unit = 'cm')
    ),
    plot.caption = element_text(
      size = 10, 
      margin = margin(b = 0.25, unit = 'cm')
    ),
    legend.text = element_text(size = bar_labels_size),
    legend.title = element_text(size = 14),
    strip.text = element_text(
      hjust = 0, 
      size = month_size,
      margin = margin(b = 0.25, unit = 'cm')
    )
  ) +
  guides(
    fill = guide_colorbar(
      barwidth = unit(bar_width_cm, 'cm'),
      barheight = unit(bar_height_cm, 'cm'),
      title.position = 'top',
      title.hjust = 0,
      title.vjust = 0,
      frame.colour = labels_color
    )
  )

Add Weekday labels back in

labels_color <- 'grey30'
schedueled_color <- '#009E73'
bar_width_cm <- 15
bar_height_cm <- 0.3
font_family <- 'Fira Sans'
bar_labels_size <- 11
month_size <- 12
nudge_labels <- 0.25
labels_size <- 3

date_counts_w_labels |> 
  ggplot(aes(wday, 5 - week)) +
  geom_tile(
    aes(fill = n), 
    col = labels_color
  ) +
  geom_text(
    aes(label = day),
    nudge_x = nudge_labels,
    nudge_y = nudge_labels,
    col = labels_color,
    size = labels_size,
    family = font_family
  ) +
  facet_wrap(vars(month), ncol = 3) +
  coord_equal(expand = FALSE) +
  scale_fill_gradient(
    high = schedueled_color, 
    low = colorspace::lighten(schedueled_color, 0.9),
  ) +
  theme_void() +
  labs(
    title = 'On Saturdays, less flights leave NYC',
    subtitle = 'Based on 336,776 schedueled flights in 2013',
    fill = 'No. of schedueled flights',
    caption = 'Data: {nycflights13} R package | Graphic: Dr. Albert Rapp'
  ) +
  theme(
    legend.position = 'top',
    text = element_text(
      color = labels_color, 
      family = font_family
    ),
    plot.title = element_text(
      size = 24, 
      margin = margin(t = 0.25, b = 0.25, unit = 'cm')
    ),
    plot.subtitle = element_text(
      size = 16, 
      margin = margin(b = 0.5, unit = 'cm')
    ),
    plot.caption = element_text(
      size = 10, 
      margin = margin(b = 0.25, unit = 'cm')
    ),
    legend.text = element_text(size = bar_labels_size),
    legend.title = element_text(size = 14),
    strip.text = element_text(
      hjust = 0, 
      size = month_size,
      margin = margin(b = 0.25, unit = 'cm')
    ),
    axis.text.x = element_text(
      margin = margin(t = -0.6, b = 0.3, unit = 'cm')
    )
  ) +
  guides(
    fill = guide_colorbar(
      barwidth = unit(bar_width_cm, 'cm'),
      barheight = unit(bar_height_cm, 'cm'),
      title.position = 'top',
      title.hjust = 0,
      title.vjust = 0,
      frame.colour = labels_color
    )
  )

Text Styling With ggplot2

Albert Rapp — Sat, 20 Jul 2024 22:00:00 GMT

In today’s blog post, we are figuring out how to fully control the text styling of the texts that we put into our ggplots. This means that we will learn

how to dynamically adjust the text color depending on the background color, and
how to use the extensive styling capabilities that the brand-new {marquee} package gives you.

Here, you will find all of the code chunks split into sections. For detailed explantions, check out the corresponding YT video:

Customize the text color based on the background color

library(tidyverse)
dat <- tibble(value = 1:5) |> 
  mutate(
    text_color = if_else(
      value <= 3,
      'black',
      'white'
    )
  ) 

dat |> 
  ggplot(aes(x = value, y = 1)) +
  geom_tile(
    aes(fill = value),
    width = 0.5, 
    height = 0.5,
    col = 'black'
  ) +
  geom_text(
    aes(label = value),
    color = dat$text_color,
    size = 8,
    fontface = 'bold',
    family = 'Source Sans Pro'
  ) +
  coord_fixed() +
  scale_fill_gradient(low = 'white', high = 'firebrick4') +
  theme_void() +
  theme(legend.position = 'none')

Dynamic text color with categorical labels

dat <- tibble(x = 1:5, letter = letters[1:5]) |> 
  mutate(
    text_color = if_else(
      letter %in% c('a', 'd'),
      'white',
      'black'
    )
  )

dat |> 
  ggplot(aes(x = x, y = 1)) +
  geom_tile(
    aes(fill = letter),
    width = 0.5, 
    height = 0.5,
    col = 'black'
  ) +
  geom_text(
    aes(label = letter),
    size = 8,
    color = dat$text_color,
    fontface = 'bold',
    family = 'Source Sans Pro'
  ) +
  coord_fixed() +
  theme_void() +
  scale_fill_brewer(palette = 'Set1') +
  theme(legend.position = 'none')

Use `geom_marquee()` instead of `geom_text()`

geom_marquee() is a drop-in replacement for geom_text() and geom_label(). Important caveat: In order for everything to render properly, you might have to update your ragg package

library(marquee)

md_text <- 'This is a **bold word** written in _Markdown_.'

tibble(x = 1, y = 1, label = md_text) |> 
  ggplot(aes(x, y)) +
  geom_marquee(
    aes(label = label),
    size = 13,
    family = 'Source Sans Pro'
  ) +
  theme_void()

Add more styles using the `style` aesthetic

tibble(x = 1, y = 1, label = md_text) |> 
  ggplot(aes(x, y)) +
  geom_marquee(
    aes(label = label),
    size = 13,
    family = 'Source Sans Pro',
    style = classic_style(
      weight = 'thin'
    )
  ) +
  theme_void()

Modify paragraph styles

my_own_style <- classic_style(
  weight = 'thin'
) |> 
  modify_style(
    'p', 
    background = 'dodgerblue4',
    padding = trbl(10),
    color = 'white',
    border_radius = 4
  )

tibble(x = 1, y = 1, label = md_text) |> 
  ggplot(aes(x, y)) +
  geom_marquee(
    aes(label = label),
    size = 13,
    family = 'Source Sans Pro',
    style = my_own_style
  ) +
  theme_void()

Modify Code styles

md_text <- 'Now let\'s try some `code` stuff and a [url]().'

tibble(x = 1, y = 1, label = md_text) |> 
  ggplot(aes(x, y)) +
  geom_marquee(
    aes(label = label),
    size = 13,
    family = 'Source Sans Pro',
    style = classic_style() |> 
      modify_style(
        'code',
        weight = 'bold',
        background = colorspace::lighten('dodgerblue4', 0.9),
        border_radius = 4,
        color = 'dodgerblue4',
        family = 'IBM Plex Mono',
        padding = trbl(0, 4, 0, 4)
      )
  ) +
  theme_void()

Use long texts as part of plot titles

md_text <- '# This is a headline

And the rest is just a regular text, i.e. paragraph, that will contain long and lengthy but also **SUPER** important information. Isn\'t that just great?'


headline_style <- classic_style() |> 
  remove_style('h1') |> 
  modify_style(
    'h1',
    weight = 'bold',
    size = 32,
    margin = trbl(b = 4),
    family = 'Merriweather'
  ) |> 
  modify_style(
    'p',
    lineheight = 1
  )

tibble(x = 1, y = 1) |> 
  ggplot(aes(x, y)) +
  geom_point(size = 10) +
  labs(title = md_text) +
  theme_minimal(
    base_size = 18, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    plot.title = element_marquee(
      width = 1,
      style = headline_style
    )
  )

Create text boxes using the `width` argument

text_box_style <- headline_style |> 
      modify_style(
        'p', 
        padding = trbl(l = 4),
        size = 16
      ) |> 
      modify_style(
        'body',
        border_radius = 4,
        border = 'dodgerblue4',
        border_size = trbl(2),
        padding = trbl(5)
      ) |> 
      modify_style(
        'h1',
        border_size = trbl(0)
      ) 

tibble(x = 1, y = 1) |> 
  ggplot(aes(x, y)) +
  geom_point(size = 10) +
  annotate(
    'marquee',
    x = 1.2,
    y = 1.5,
    label = md_text,
    width = 0.4,
    hjust = 0,
    fill = colorspace::lighten('dodgerblue1', 0.7),
    style = text_box_style
  ) +
  labs(title = md_text) +
  theme_minimal(
    base_size = 18, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    plot.title = element_marquee(
      width = 1,
      style = headline_style
    )
  ) +
  coord_cartesian(
    xlim = c(0, 2),
    ylim = c(0, 2)
  )

Colorize single words in your title

md_text <- '# This is a headline

And the rest is just a regular text, i.e. paragraph, that will contain long and lengthy but also {.red **SUPER** important information}. Isn\'t that just great?'

tibble(x = 1, y = 1) |> 
  ggplot(aes(x, y)) +
  geom_point(size = 10) +
  annotate(
    'marquee',
    x = 1.2,
    y = 1.5,
    label = md_text,
    width = 0.4,
    hjust = 0,
    fill = colorspace::lighten('dodgerblue1', 0.7),
    style = text_box_style
  ) +
  labs(title = md_text) +
  theme_minimal(
    base_size = 18, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    plot.title = element_marquee(
      width = 1,
      style = headline_style
    )
  ) +
  coord_cartesian(
    xlim = c(0, 2),
    ylim = c(0, 2)
  )

Define your own inline style

md_text <- '# This is a headline

And the rest is just a regular text, i.e. paragraph, that will contain long and lengthy but also {.my_style **SUPER** important information}. Isn\'t that just great?'

tibble(x = 1, y = 1) |> 
  ggplot(aes(x, y)) +
  geom_point(size = 10) +
  annotate(
    'marquee',
    x = 1.2,
    y = 1.5,
    label = md_text,
    width = 0.4,
    hjust = 0,
    fill = colorspace::lighten('dodgerblue1', 0.7),
    style = text_box_style |> 
      modify_style(
        'my_style',
        color = 'seagreen'
      )
  ) +
  labs(title = md_text) +
  theme_minimal(
    base_size = 18, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    plot.title = element_marquee(
      width = 1,
      style = headline_style |> 
        modify_style(
          'my_style',
          color = 'seagreen'
        )
    )
  ) +
  coord_cartesian(
    xlim = c(0, 2),
    ylim = c(0, 2)
  )

Quick dataViz techniques for nicer line charts with ggplot

Albert Rapp — Sat, 06 Jul 2024 22:00:00 GMT

Line charts are one of the most fundamental chart type out there. That’s why there’s a lot of tips for line charts out there. Today, I’m going to walk you through a couple of techniques that you can use to make your line chart nicer. Here, I’ll provide you with all the code chunks. All explanations can be found in my corresponding YT video:

Get the data

library(tidyverse)
sp500_data_wide <- gt::sp500 |>
  select(date, open, close) |> 
  filter(year(date) == 2014, month(date) == 1)
sp500_data_wide
## # A tibble: 21 × 3
##    date        open close
##          
##  1 2014-01-31 1791. 1783.
##  2 2014-01-30 1777. 1794.
##  3 2014-01-29 1790. 1774.
##  4 2014-01-28 1783  1792.
##  5 2014-01-27 1791. 1782.
##  6 2014-01-24 1827. 1790.
##  7 2014-01-23 1842. 1828.
##  8 2014-01-22 1845. 1845.
##  9 2014-01-21 1841. 1844.
## 10 2014-01-17 1844. 1839.
## # ℹ 11 more rows

Bring into a nice format for ggplot

sp500_data <- sp500_data_wide |> 
  pivot_longer(
    cols = -date,
    names_to = 'type',
    values_to = 'price'
  )

Create a basic line chart

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25)

Apply a theme

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank()
  )

Use nicer colors

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank()
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  )

Use meaningful labels

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank()
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  )

Make labels into currency

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank()
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  )

Use direct labels instead of legend

sp500_data |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  geom_text(
    data = sp500_data |> slice_head(n = 1, by = type),
    aes(label = type),
    hjust = 0,
    vjust = 0,
    family = 'Source Sans Pro',
    size = 10,
    nudge_x = 0.1
  ) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank(),
    legend.position = 'none'
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  ) +
  scale_x_date(
    limits = c(
      make_date(2014, 1, 1), 
      make_date(2014, 2 ,3)
    )
  )

Make nicer labels

sp500_data_with_nicer_labels <- sp500_data |> 
  mutate(
    type = if_else(
      type == 'open',
      'Opening price',
      'Closing price'
    )
  ) 

sp500_data_with_nicer_labels|> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  geom_text(
    data = sp500_data_with_nicer_labels |> 
      slice_head(n = 1, by = type),
    aes(label = type),
    hjust = 0,
    vjust = 0,
    family = 'Source Sans Pro',
    size = 10,
    nudge_x = 0.1
  ) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank(),
    legend.position = 'none'
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  ) +
  scale_x_date(
    limits = c(
      make_date(2014, 1, 1), 
      make_date(2014, 2, 8)
    )
  )

Place labels closer to the lines

sp500_data_with_nicer_labels |> 
  ggplot(aes(date, price, col = type)) +
  geom_line(linewidth = 1.25) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Opening price'),
    aes(label = type),
    hjust = 0.76,
    vjust = 0,
    family = 'Source Sans Pro',
    size = 8
  ) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Closing price'),
    aes(label = type),
    hjust = 0.77,
    vjust = 1,
    family = 'Source Sans Pro',
    size = 8,
    text_smoothing = 40,
    offset = unit(-14, 'mm')
  ) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank(),
    legend.position = 'none'
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  )

Highlight area between lines.

sp500_data_with_nicer_labels |> 
  ggplot(aes(date, price, col = type)) +
  ggbraid::geom_braid(
    data = sp500_data_wide, 
    aes(
      y = NULL, ## Overwrite the inherited aes from ggplot()
      col = NULL, 
      ymin = open, 
      ymax = close, 
      fill = open < close
    ), 
    alpha = 0.6
  ) +
  geom_line(linewidth = 1.25) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Opening price'),
    aes(label = type),
    hjust = 0.76,
    vjust = 0,
    family = 'Source Sans Pro',
    size = 8
  ) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Closing price'),
    aes(label = type),
    hjust = 0.77,
    vjust = 1,
    family = 'Source Sans Pro',
    size = 8,
    text_smoothing = 40,
    offset = unit(-14, 'mm')
  ) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank(),
    legend.position = 'none'
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  scale_fill_manual(
    values = c('TRUE' = '#0072B2', 'FALSE' = '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  ) 
## `geom_braid()` using method = 'line'

Add a callout label box

sp500_data_with_nicer_labels |> 
  ggplot(aes(date, price, col = type)) +
  ggbraid::geom_braid(
    data = sp500_data_wide, 
    aes(
      y = NULL, ## Overwrite the inherited aes from ggplot()
      col = NULL, 
      ymin = open, 
      ymax = close, 
      fill = open < close
    ), 
    alpha = 0.6
  ) +
  geom_line(linewidth = 1.25) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Opening price'),
    aes(label = type),
    hjust = 0.76,
    vjust = 0,
    family = 'Source Sans Pro',
    size = 8
  ) +
  geomtextpath::geom_textline(
    data = sp500_data_with_nicer_labels |> 
      filter(type == 'Closing price'),
    aes(label = type),
    hjust = 0.77,
    vjust = 1,
    family = 'Source Sans Pro',
    size = 8,
    text_smoothing = 40,
    offset = unit(-14, 'mm')
  ) +
  ggforce::geom_mark_circle(
    data = tibble(
      date = make_date(2014, 1, 24),
      price = 1820
    ),
    aes(
      col = NULL, 
      label = 'This area signals whether the\nclosing price or opening price was\nhigher on a given day'
    ),
    fill = 'white',
    color = 'grey20',
    alpha = 1,
    x0 = make_date(2014, 1, 13),
    y0 = 1805,
    label.family = 'Source Sans Pro',
    label.colour = 'grey20',
    label.hjust = 0,
    label.fontsize = 12,
    label.fontface = 'plain',
    con.colour = 'grey20',
    con.cap = unit(1, 'mm'),
    expand = 0.011
  ) +
  theme_minimal(
    base_size = 20, 
    base_family = 'Source Sans Pro'
  ) +
  theme(
    panel.grid.minor = element_blank(),
    legend.position = 'none'
  ) +
  scale_color_manual(
    values = c('#0072B2', '#D55E00')
  ) +
  scale_fill_manual(
    values = c('TRUE' = '#0072B2', 'FALSE' = '#D55E00')
  ) +
  labs(
    x = element_blank(), 
    y = element_blank(),
    title = 'SP500 Prices in January 2014'
  ) +
  scale_y_continuous(
    labels = scales::label_dollar()
  ) 
## `geom_braid()` using method = 'line'

How to avoid empty line charts

Albert Rapp — Sat, 29 Jun 2024 22:00:00 GMT

Have you ever tried to create a line chart with ggplot only to find that the chart remains blank and a warning showing up that each group consists out of one observation? That’s a common struggle and I used to attempt a lot of trial and error to fix this. In today’s blog post we’re going to take a look what’s going on when this warning message occurs and how to fix that like a pro. Like always, you can find a video version of this blog post on YouTube:

Generate some fake data

First, what we need is data to work with. Here, I’m just going to simulate a bit of random data. And at the end of the day, the exact data isn’t that important. Just know that all of the data sets I use in this blog post are simulated.

You can find the code for that simulation in this folded code chunk. Feel free to check that out if you’re curious.

Code

Master Data Cleaning & Get To Insights Faster

Albert Rapp — Mon, 17 Jun 2024 22:00:00 GMT

Guten Tag! 👋

As data scientists we are judged by how quickly we can generate meaningful insights. But we’re often stuck with cleaning our data first. That’s bad. So that’s why I’m super excited to tell you that my new “Data Cleaning with R Master Class” just launched.

This video course will teach you how to stop wasting time and clean your messy data fast. In fact, I will give you an in-depth curriculum for all your data cleaning needs.

Count me in!

What you can get from the course

Once you complete the course, I am confident that you can leave that data cleaning mess behind you much quicker and get to your insights faster. Now, how do I plan to teach you how to do that?

I’ve condensed years of experience into five parts with each of them focusing on one specific aspect of your data cleaning efforts. All of these parts first show you the mechanics of useful data functions and then tie everything together with real-world examples.

Here’s the breakdown of the course:

Part 1: Get a deep understanding of the basics of computing and summarizing columns. Improve your productivity with advanced tricks.
Part 2: Learn how to read any file and get it from a messy state into a usable format. This will be particularly useful for Excel files.
Part 3: Knowing how to work with text data (and regex in particular) unlocks so many doors for you. I’ll show you how that works.
Part 4: Working with times and dates is always painful. But your live becomes easier when you use powerful functions from {lubridate}.
Part 5: Unlocking the hard but insanely useful functional programming paradigm will make you so much more productive.

Sign me up!

Where is the special offer ?

I know what you’re thinking. A course release is great. But there needs to be some promo offer, right? And you’re right. I’ve got a deal in store for you.

First, this course is currently in pre-sale mode. This means that you get full access to

Part 1 of the course now and
Parts 2, 3, 4 & 5 as they become available in the next couple of months.

And the best part: The course is currently priced at 50% off compared to the final price when the course is completed.

Take me to the course!

This course has a jam-packed & ambitious learning path for you. Completing it will make you so much more efficient at data cleaning. So, if you want to get to your data insights quicker, join the course today. 🎉

The 6 Most Fundamental Functions for Data Cleaning with R

Albert Rapp — Sat, 15 Jun 2024 22:00:00 GMT

In this blog post I’m going to show you the six most fundamental functions for your data cleaning journey. And as always, you can watch the video version of this blog post on YouTube:

Getting started

All of these functions come from the tidyverse. So this is why we are going to load that package.

library(tidyverse)

And of course, we’re going to need some data, and this is where the palmerpenguins package comes in. We can either load the palmerpenguins package like we did with the tidyverse, or we can just access the penguins data set via its full name, namely the palmerpenguins package name followed by :: followed by the data set name.

penguins <- palmerpenguins::penguins
penguins
## # A tibble: 344 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                  
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 334 more rows
## # ℹ 2 more variables: sex , year

Count all the things with `count()`

Once we have that data set, we can pass it to the count() function. Once we do that, we see in the console that we get a count.

penguins |> 
  count()
## # A tibble: 1 × 1
##       n
##   
## 1   344

But this output right now isn’t particularly useful because it just shows us how many things are in this data set. You see, count() becomes way more useful when you also add a column name from the dataset inside of count(). That way, we could count how many different species are in the data set.

penguins |> 
  count(species)
## # A tibble: 3 × 2
##   species       n
##        
## 1 Adelie      152
## 2 Chinstrap    68
## 3 Gentoo      124

So that’s how we know how many Adelie, Gentoo and Chinstrap penguins are there. Furthermore, we can also sort the rows in this data set by also adding the sort = TRUE argument inside of count(). This is particularly helpful if you have a data set that has a whole bunch of different things inside one column and you want to get the most frequent ones at the top.

penguins |> 
  count(species, sort = TRUE)
## # A tibble: 3 × 2
##   species       n
##        
## 1 Adelie      152
## 2 Gentoo      124
## 3 Chinstrap    68

And we could even count combinations of different columns.

penguins |> 
  count(species, island, sort = TRUE)
## # A tibble: 5 × 3
##   species   island        n
##             
## 1 Gentoo    Biscoe      124
## 2 Chinstrap Dream        68
## 3 Adelie    Dream        56
## 4 Adelie    Torgersen    52
## 5 Adelie    Biscoe       44

Let me note that count() is super useful because obviously you can count things, but also I use it all of the time as intermediate steps when I filter or rearrange some part of the data. This helps me to check that the count of what is now inside of a data set matches what I expect after some data operation.

Get the right columns with `select()`

Next let us talk about the select() function. This function helps you select columns from your data set. Say, you have a data set with a whole bunch of columns and you want to focus on a subset of that. Then you can just

take your data,
pass it to the select() function, and
list all of the things that you want to select.

penguins |> 
  select(flipper_length_mm, bill_length_mm, species)
## # A tibble: 344 × 3
##    flipper_length_mm bill_length_mm species
##                             
##  1               181           39.1 Adelie 
##  2               186           39.5 Adelie 
##  3               195           40.3 Adelie 
##  4                NA           NA   Adelie 
##  5               193           36.7 Adelie 
##  6               190           39.3 Adelie 
##  7               181           38.9 Adelie 
##  8               195           39.2 Adelie 
##  9               193           34.1 Adelie 
## 10               190           42   Adelie 
## # ℹ 334 more rows

That’s was pretty easy, right? Here, I kind of want to leave it at this stage. But be aware that there are a whole bunch of advanced tricks that you can use in select() to not have to list all of the things.

Instead, you can just describe the things that you want. For example, you can get all the columns

that have a particular text in their name (like _mm in the penguins dataset) or
that are of specific type (e.g. numeric)

In these cases, you can use a couple of tricks in combination with select() that make your life much easier. But here I really want to focus on the fundamentals. So let’s ignore those advanced tricks for now.

Reduce the rows to specific observations with `filter()`

The next essential function is filter(). It is like select() but for rows. Using that function you can filter your data according to specific criteria. That way, you get all of the columns of the data that you pass to filter(), but you get only the rows that match a specific criteria.

For example, we can

take our penguins data,
pass it to the filter() function, and
state that we want to get only the penguins whose bill length is larger than 55 millimeters.

penguins |> 
  filter(bill_length_mm > 55)
## # A tibble: 5 × 8
##   species   island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                
## 1 Gentoo    Biscoe           59.6          17                 230        6050
## 2 Gentoo    Biscoe           55.9          17                 228        5600
## 3 Gentoo    Biscoe           55.1          16                 230        5850
## 4 Chinstrap Dream            58            17.8               181        3700
## 5 Chinstrap Dream            55.8          19.8               207        4000
## # ℹ 2 more variables: sex , year

I’ll explain how to read that code in a second (in case you don’t find it intuitive.) Let me add one more example first, though. For instance, we could also use the between() function in combination with filter() to find the penguins that have a bill length larger than 55 but smaller than 58.

penguins |> 
  filter(between(bill_length_mm, 55, 58))
## # A tibble: 4 × 8
##   species   island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##                                                
## 1 Gentoo    Biscoe           55.9          17                 228        5600
## 2 Gentoo    Biscoe           55.1          16                 230        5850
## 3 Chinstrap Dream            58            17.8               181        3700
## 4 Chinstrap Dream            55.8          19.8               207        4000
## # ℹ 2 more variables: sex , year

Okay, here you might wonder how to understand this thing here. So, let me give you an insight of how filter() function does things.

Basically, whenever you describe a condition inside of filter(), then what happens is that it takes the column that you used and applies the command to get a vector full of TRUE and FALSE. Let’s try this out manually to see what happens. Let’s grab the bill lengths.

bill_length_mm
## Error in eval(expr, envir, enclos): object 'bill_length_mm' not found

As you can see, we can’t just access the data column via its name outside of the filter() function. R simply doesn’t know where this column name is coming from. Inside of the filter() function R knows to look at the data set. But outside of filter() you will have to tell R that this thing comes from our penguins data set. You do that by via the data set name and the $ operator.

penguins$bill_length_mm
##   [1] 39.1 39.5 40.3   NA 36.7 39.3 38.9 39.2 34.1 42.0 37.8 37.8 41.1 38.6 34.6
##  [16] 36.6 38.7 42.5 34.4 46.0 37.8 37.7 35.9 38.2 38.8 35.3 40.6 40.5 37.9 40.5
##  [31] 39.5 37.2 39.5 40.9 36.4 39.2 38.8 42.2 37.6 39.8 36.5 40.8 36.0 44.1 37.0
##  [46] 39.6 41.1 37.5 36.0 42.3 39.6 40.1 35.0 42.0 34.5 41.4 39.0 40.6 36.5 37.6
##  [61] 35.7 41.3 37.6 41.1 36.4 41.6 35.5 41.1 35.9 41.8 33.5 39.7 39.6 45.8 35.5
##  [76] 42.8 40.9 37.2 36.2 42.1 34.6 42.9 36.7 35.1 37.3 41.3 36.3 36.9 38.3 38.9
##  [91] 35.7 41.1 34.0 39.6 36.2 40.8 38.1 40.3 33.1 43.2 35.0 41.0 37.7 37.8 37.9
## [106] 39.7 38.6 38.2 38.1 43.2 38.1 45.6 39.7 42.2 39.6 42.7 38.6 37.3 35.7 41.1
## [121] 36.2 37.7 40.2 41.4 35.2 40.6 38.8 41.5 39.0 44.1 38.5 43.1 36.8 37.5 38.1
## [136] 41.1 35.6 40.2 37.0 39.7 40.2 40.6 32.1 40.7 37.3 39.0 39.2 36.6 36.0 37.8
## [151] 36.0 41.5 46.1 50.0 48.7 50.0 47.6 46.5 45.4 46.7 43.3 46.8 40.9 49.0 45.5
## [166] 48.4 45.8 49.3 42.0 49.2 46.2 48.7 50.2 45.1 46.5 46.3 42.9 46.1 44.5 47.8
## [181] 48.2 50.0 47.3 42.8 45.1 59.6 49.1 48.4 42.6 44.4 44.0 48.7 42.7 49.6 45.3
## [196] 49.6 50.5 43.6 45.5 50.5 44.9 45.2 46.6 48.5 45.1 50.1 46.5 45.0 43.8 45.5
## [211] 43.2 50.4 45.3 46.2 45.7 54.3 45.8 49.8 46.2 49.5 43.5 50.7 47.7 46.4 48.2
## [226] 46.5 46.4 48.6 47.5 51.1 45.2 45.2 49.1 52.5 47.4 50.0 44.9 50.8 43.4 51.3
## [241] 47.5 52.1 47.5 52.2 45.5 49.5 44.5 50.8 49.4 46.9 48.4 51.1 48.5 55.9 47.2
## [256] 49.1 47.3 46.8 41.7 53.4 43.3 48.1 50.5 49.8 43.5 51.5 46.2 55.1 44.5 48.8
## [271] 47.2   NA 46.8 50.4 45.2 49.9 46.5 50.0 51.3 45.4 52.7 45.2 46.1 51.3 46.0
## [286] 51.3 46.6 51.7 47.0 52.0 45.9 50.5 50.3 58.0 46.4 49.2 42.4 48.5 43.2 50.6
## [301] 46.7 52.0 50.5 49.5 46.4 52.8 40.9 54.2 42.5 51.0 49.7 47.5 47.6 52.0 46.9
## [316] 53.5 49.0 46.2 50.9 45.5 50.9 50.8 50.1 49.0 51.5 49.8 48.1 51.4 45.7 50.7
## [331] 42.5 52.2 45.2 49.3 50.2 45.6 51.9 46.8 45.7 55.8 43.5 49.6 50.8 50.2

So now you see that we get a vector full of the length that we want to look at. Now, what happens inside of filter() is that this vector is applied according to what you specified. In the first example, we just checked is this thing larger than 55. Here’s what that results in.

penguins$bill_length_mm > 55
##   [1] FALSE FALSE FALSE    NA FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [49] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [61] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [73] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [85] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [97] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [109] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [121] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [133] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [145] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [157] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [169] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [181] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
## [193] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [205] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [217] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [229] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [241] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [253] FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [265] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE    NA FALSE FALSE FALSE FALSE
## [277] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [289] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
## [301] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [313] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [325] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [337] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE

Now you see that what you get is a vector full of TRUEs and FALSEs. And as you can probably guess:

TRUE corresponds to the occasions where the bill length was indeed larger than 55, and
FALSE corresponds to the cases where this wasn’t the case.

Really, what you put inside of filter() are operations that at the end of the day deliver TRUEs and FALSEs. Then, filter() just takes the rows that correspond to TRUE. You could simulate this by taking these TRUEs and FALSEs that we’ve just captured manually and sticking that into the which() function.

which(penguins$bill_length_mm > 55)
## [1] 186 254 268 294 340

This gives you all of the row numbers for which you had a TRUE, and these are the rows that filter() will return.

Now why did I explain this in so much detail when it was really a simple operation? The thing is, I want you to understand this as things become more complicated. For example, with our second example using the between() function, you might wonder why this was valid syntax. Basically, you have to understand what between() does. So here’s an example.

between(c(54, 55, 56, 57, 58), 55, 57)
## [1] FALSE  TRUE  TRUE  TRUE FALSE

As you can see, between() is a function that does exactly what we just talked about, i.e. it returns TRUEs and FALSEs. So really, this was just a different operation that has some slightly other syntax. But the syntax doesn’t matter to us as long as at the end of the day this operation that we use inside of filter() will return TRUEs and FALSEs so that filter() can do its thing.

Calculating and transforming columns with `mutate()`

Now let’s compute new things from our data. For example, we can compute a new column that takes the bill lengths that we’ve just looked at and scales them. Scaling is a technique used to transform the values of a variable so that they are centered around zero. This transformation allows you to get a better understanding of whether a value is high or low relative to the other values in the dataset (even if you are not familiar with the specific units of measurement.)

For example, if you scale the bill length variable, a scaled value of

0 would indicate an average bill length,
while a positive value would indicate an above-average bill length, and
a negative value would indicate a below-average bill length.

And to do this scaling, we need the mutate() function. It allows you to modify existing columns or create new columns in a dataset. Here’s how it works:

Pass your data set to the mutate() function.
Specify the column name and set it equal to the desired transformation.

penguins |> 
  mutate(bill_length_mm = scale(bill_length_mm)[,1]) |> 
  select(bill_length_mm)
## # A tibble: 344 × 1
##    bill_length_mm
##             
##  1         -0.883
##  2         -0.810
##  3         -0.663
##  4         NA    
##  5         -1.32 
##  6         -0.847
##  7         -0.920
##  8         -0.865
##  9         -1.80 
## 10         -0.352
## # ℹ 334 more rows

And via the same strategy, we can also create a new column. We just have to assign a new column name.

penguins |> 
  mutate(scaled_bill_length = scale(bill_length_mm)[,1]) |> 
  select(bill_length_mm, scaled_bill_length)
## # A tibble: 344 × 2
##    bill_length_mm scaled_bill_length
##                           
##  1           39.1             -0.883
##  2           39.5             -0.810
##  3           40.3             -0.663
##  4           NA               NA    
##  5           36.7             -1.32 
##  6           39.3             -0.847
##  7           38.9             -0.920
##  8           39.2             -0.865
##  9           34.1             -1.80 
## 10           42               -0.352
## # ℹ 334 more rows

Behind the scenes, mutate() takes the specified column…

penguins$bill_length_mm
##   [1] 39.1 39.5 40.3   NA 36.7 39.3 38.9 39.2 34.1 42.0 37.8 37.8 41.1 38.6 34.6
##  [16] 36.6 38.7 42.5 34.4 46.0 37.8 37.7 35.9 38.2 38.8 35.3 40.6 40.5 37.9 40.5
##  [31] 39.5 37.2 39.5 40.9 36.4 39.2 38.8 42.2 37.6 39.8 36.5 40.8 36.0 44.1 37.0
##  [46] 39.6 41.1 37.5 36.0 42.3 39.6 40.1 35.0 42.0 34.5 41.4 39.0 40.6 36.5 37.6
##  [61] 35.7 41.3 37.6 41.1 36.4 41.6 35.5 41.1 35.9 41.8 33.5 39.7 39.6 45.8 35.5
##  [76] 42.8 40.9 37.2 36.2 42.1 34.6 42.9 36.7 35.1 37.3 41.3 36.3 36.9 38.3 38.9
##  [91] 35.7 41.1 34.0 39.6 36.2 40.8 38.1 40.3 33.1 43.2 35.0 41.0 37.7 37.8 37.9
## [106] 39.7 38.6 38.2 38.1 43.2 38.1 45.6 39.7 42.2 39.6 42.7 38.6 37.3 35.7 41.1
## [121] 36.2 37.7 40.2 41.4 35.2 40.6 38.8 41.5 39.0 44.1 38.5 43.1 36.8 37.5 38.1
## [136] 41.1 35.6 40.2 37.0 39.7 40.2 40.6 32.1 40.7 37.3 39.0 39.2 36.6 36.0 37.8
## [151] 36.0 41.5 46.1 50.0 48.7 50.0 47.6 46.5 45.4 46.7 43.3 46.8 40.9 49.0 45.5
## [166] 48.4 45.8 49.3 42.0 49.2 46.2 48.7 50.2 45.1 46.5 46.3 42.9 46.1 44.5 47.8
## [181] 48.2 50.0 47.3 42.8 45.1 59.6 49.1 48.4 42.6 44.4 44.0 48.7 42.7 49.6 45.3
## [196] 49.6 50.5 43.6 45.5 50.5 44.9 45.2 46.6 48.5 45.1 50.1 46.5 45.0 43.8 45.5
## [211] 43.2 50.4 45.3 46.2 45.7 54.3 45.8 49.8 46.2 49.5 43.5 50.7 47.7 46.4 48.2
## [226] 46.5 46.4 48.6 47.5 51.1 45.2 45.2 49.1 52.5 47.4 50.0 44.9 50.8 43.4 51.3
## [241] 47.5 52.1 47.5 52.2 45.5 49.5 44.5 50.8 49.4 46.9 48.4 51.1 48.5 55.9 47.2
## [256] 49.1 47.3 46.8 41.7 53.4 43.3 48.1 50.5 49.8 43.5 51.5 46.2 55.1 44.5 48.8
## [271] 47.2   NA 46.8 50.4 45.2 49.9 46.5 50.0 51.3 45.4 52.7 45.2 46.1 51.3 46.0
## [286] 51.3 46.6 51.7 47.0 52.0 45.9 50.5 50.3 58.0 46.4 49.2 42.4 48.5 43.2 50.6
## [301] 46.7 52.0 50.5 49.5 46.4 52.8 40.9 54.2 42.5 51.0 49.7 47.5 47.6 52.0 46.9
## [316] 53.5 49.0 46.2 50.9 45.5 50.9 50.8 50.1 49.0 51.5 49.8 48.1 51.4 45.7 50.7
## [331] 42.5 52.2 45.2 49.3 50.2 45.6 51.9 46.8 45.7 55.8 43.5 49.6 50.8 50.2

…performs the requested transformation,

scale(penguins$bill_length_mm)[,1]
##   [1] -0.88320467 -0.80993901 -0.66340769          NA -1.32279862 -0.84657184
##   [7] -0.91983750 -0.86488825 -1.79902541 -0.35202864 -1.12131806 -1.12131806
##  [13] -0.51687637 -0.97478674 -1.70744334 -1.34111504 -0.95647033 -0.26044656
##  [19] -1.74407616  0.38062795 -1.12131806 -1.13963448 -1.46932994 -1.04805240
##  [25] -0.93815391 -1.57922843 -0.60845845 -0.62677486 -1.10300165 -0.62677486
##  [31] -0.80993901 -1.23121655 -0.80993901 -0.55350920 -1.37774787 -0.86488825
##  [37] -0.93815391 -0.31539581 -1.15795089 -0.75498976 -1.35943145 -0.57182562
##  [43] -1.45101353  0.03261607 -1.26784938 -0.79162259 -0.51687637 -1.17626731
##  [49] -1.45101353 -0.29707939 -0.79162259 -0.70004052 -1.63417768 -0.35202864
##  [55] -1.72575975 -0.46192713 -0.90152108 -0.60845845 -1.35943145 -1.15795089
##  [61] -1.50596277 -0.48024354 -1.15795089 -0.51687637 -1.37774787 -0.42529430
##  [67] -1.54259560 -0.51687637 -1.46932994 -0.38866147 -1.90892390 -0.77330618
##  [73] -0.79162259  0.34399512 -1.54259560 -0.20549732 -0.55350920 -1.23121655
##  [79] -1.41438070 -0.33371222 -1.70744334 -0.18718091 -1.32279862 -1.61586126
##  [85] -1.21290014 -0.48024354 -1.39606428 -1.28616579 -1.02973599 -0.91983750
##  [91] -1.50596277 -0.51687637 -1.81734182 -0.79162259 -1.41438070 -0.57182562
##  [97] -1.06636882 -0.66340769 -1.98218956 -0.13223166 -1.63417768 -0.53519279
## [103] -1.13963448 -1.12131806 -1.10300165 -0.77330618 -0.97478674 -1.04805240
## [109] -1.06636882 -0.13223166 -1.06636882  0.30736229 -0.77330618 -0.31539581
## [115] -0.79162259 -0.22381374 -0.97478674 -1.21290014 -1.50596277 -0.51687637
## [121] -1.41438070 -1.13963448 -0.68172411 -0.46192713 -1.59754485 -0.60845845
## [127] -0.93815391 -0.44361071 -0.90152108  0.03261607 -0.99310316 -0.15054808
## [133] -1.30448221 -1.17626731 -1.06636882 -0.51687637 -1.52427919 -0.68172411
## [139] -1.26784938 -0.77330618 -0.68172411 -0.60845845 -2.16535371 -0.59014203
## [145] -1.21290014 -0.90152108 -0.86488825 -1.34111504 -1.45101353 -1.12131806
## [151] -1.45101353 -0.44361071  0.39894437  1.11328455  0.87517115  1.11328455
## [157]  0.67369059  0.47221003  0.27072946  0.50884286 -0.11391525  0.52715927
## [163] -0.55350920  0.93012040  0.28904588  0.82022191  0.34399512  0.98506964
## [169] -0.35202864  0.96675323  0.41726078  0.87517115  1.14991738  0.21578022
## [175]  0.47221003  0.43557720 -0.18718091  0.39894437  0.10588173  0.71032342
## [181]  0.78358908  1.11328455  0.61874135 -0.20549732  0.21578022  2.87166037
## [187]  0.94843681  0.82022191 -0.24213015  0.08756532  0.01429966  0.87517115
## [193] -0.22381374  1.04001889  0.25241305  1.04001889  1.20486662 -0.05896600
## [199]  0.28904588  1.20486662  0.17914739  0.23409663  0.49052644  0.83853832
## [205]  0.21578022  1.13160096  0.47221003  0.19746381 -0.02233317  0.28904588
## [211] -0.13223166  1.18655021  0.25241305  0.41726078  0.32567871  1.90089038
## [217]  0.34399512  1.07665172  0.41726078  1.02170247 -0.07728242  1.24149945
## [223]  0.69200701  0.45389361  0.78358908  0.47221003  0.45389361  0.85685474
## [229]  0.65537418  1.31476511  0.23409663  0.23409663  0.94843681  1.57119492
## [235]  0.63705776  1.11328455  0.17914739  1.25981586 -0.09559883  1.35139794
## [241]  0.65537418  1.49792926  0.65537418  1.51624567  0.28904588  1.02170247
## [247]  0.10588173  1.25981586  1.00338606  0.54547569  0.82022191  1.31476511
## [253]  0.83853832  2.19395302  0.60042493  0.94843681  0.61874135  0.52715927
## [259] -0.40697788  1.73604265 -0.11391525  0.76527266  1.20486662  1.07665172
## [265] -0.07728242  1.38803077  0.41726078  2.04742170  0.10588173  0.89348757
## [271]  0.60042493          NA  0.52715927  1.18655021  0.23409663  1.09496813
## [277]  0.47221003  1.11328455  1.35139794  0.27072946  1.60782775  0.23409663
## [283]  0.39894437  1.35139794  0.38062795  1.35139794  0.49052644  1.42466360
## [289]  0.56379210  1.47961284  0.36231154  1.20486662  1.16823379  2.57859773
## [295]  0.45389361  0.96675323 -0.27876298  0.83853832 -0.13223166  1.22318303
## [301]  0.50884286  1.47961284  1.20486662  1.02170247  0.45389361  1.62614416
## [307] -0.55350920  1.88257397 -0.26044656  1.29644869  1.05833530  0.65537418
## [313]  0.67369059  1.47961284  0.54547569  1.75435906  0.93012040  0.41726078
## [319]  1.27813228  0.28904588  1.27813228  1.25981586  1.13160096  0.93012040
## [325]  1.38803077  1.07665172  0.76527266  1.36971435  0.32567871  1.24149945
## [331] -0.26044656  1.51624567  0.23409663  0.98506964  1.14991738  0.30736229
## [337]  1.46129643  0.52715927  0.32567871  2.17563660 -0.07728242  1.04001889
## [343]  1.25981586  1.14991738

and then assigns that vector do the desired column name.

Reduce your data to key quantities with `summarize()`

The summarize() function in R allows you to summarize your data into single values. It works similarly to mutate(), but instead of adding new columns, it reduces the data set to a single row. For example, you can use summarize() to compute the mean of the bill_length_mm and flipper_length_mm columns

penguins |> 
  summarize(
    mean_bill_length = mean(bill_length_mm, na.rm = TRUE),
    mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE)
  )
## # A tibble: 1 × 2
##   mean_bill_length mean_flipper_length
##                             
## 1             43.9                201.

Note that we use na.rm = TRUE to remove any missing values before computing the means. And if you want to repeat the calculations for different subsets of your data, such as different species of penguins, you can use the .by argument:

penguins |> 
  summarize(
    mean_bill_length = mean(bill_length_mm, na.rm = TRUE),
    mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE),
    .by = species
  )
## # A tibble: 3 × 3
##   species   mean_bill_length mean_flipper_length
##                                  
## 1 Adelie                38.8                190.
## 2 Gentoo                47.5                217.
## 3 Chinstrap             48.8                196.

This will repeat the calculations separately for each species of penguin.

Sort your data with `arrange()`

The arrange() function in R allows you to sort the rows of your dataset based on one or more columns. By default, it sorts in ascending order.

penguins |> 
  summarize(
    mean_bill_length = mean(bill_length_mm, na.rm = TRUE),
    mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE),
    .by = species
  ) |> 
  arrange(mean_flipper_length)
## # A tibble: 3 × 3
##   species   mean_bill_length mean_flipper_length
##                                  
## 1 Adelie                38.8                190.
## 2 Chinstrap             48.8                196.
## 3 Gentoo                47.5                217.

But you can use the desc() function to sort in descending order:

penguins |> 
  summarize(
    mean_bill_length = mean(bill_length_mm, na.rm = TRUE),
    mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE),
    .by = species
  )  |> 
  arrange(desc(mean_flipper_length))
## # A tibble: 3 × 3
##   species   mean_bill_length mean_flipper_length
##                                  
## 1 Gentoo                47.5                217.
## 2 Chinstrap             48.8                196.
## 3 Adelie                38.8                190.

Conclusion

In this blog post/video, we covered the 6 most fundamental functions for data cleaning with R. These functions provide a solid foundation for transforming and manipulating data in R.

If you enjoyed this content and want to learn more advanced data cleaning techniques, be sure to check out my Data Cleaning with R Master Class. In this master class, we

dive deeper into these functions,
learn to clean specific data formats (such as Excel files),
handling text data, and
deal with date formats.

Be sure to check out the course. And if you found this helpful, here are some other ways I can help you:

3 Minute Wednesdays: A weekly newsletter with bite-sized tips and tricks for R users
Insightful Data Visualizations for “Uncreative” R Users: A course that teaches you how to leverage {ggplot2} to make charts that communicate effectively without being a design expert.

Why you shouldn’t use boxplots

Albert Rapp — Sat, 08 Jun 2024 22:00:00 GMT

Box plots are a very common tool in data visualization to show how your data is distributed. But they have a crucial flaw. Let’s find out what that flaw is.

And if you’re interested in the video version of this blog post, you can find it here:

The flaw of box plots

Imagine that you have a numeric variable, like the weight of penguins.

library(tidyverse)
penguins_dat <- palmerpenguins::penguins |> 
  filter(!is.na(sex))

penguins_dat |> 
  ggplot(aes(x = body_mass_g, y = '')) +
  geom_point(
    size = 4, alpha = 0.75,
    shape = 21,
    fill = 'dodgerblue4',
    color = 'black',
    position = position_jitter(
      height = 0.25,
      seed = 2343
    )
  ) +
  theme_minimal(
    base_size = 18, 
    base_family = 'Source Sans Pro'
  ) +
  labs(
    x = 'Weight (in g)', 
    y = element_blank(),
    title = 'Distribution of Penguin Weights'  
  ) +
  theme(
    panel.grid.minor = element_blank(),
    panel.grid.major.y = element_blank(),
    plot.title = element_text(
      size = rel(1.5),
      family = 'Merriweather',
      face = 'bold'
    )
  )

RStudio Shortcuts and Settings

Albert Rapp — Sat, 01 Jun 2024 22:00:00 GMT

RStudio is the premier way to write R code. It is very user-friendly as it’s super easy to set up. But it also has a lot of pretty advanced features. In today’s blog post I’ll give you a list of my favorite RStudio tricks and shortcuts. If you want to see detailed explanations, you can check out my video here:

RStudio Theme Setting

Use a dark theme for better visibility
Enable rainbow parentheses and rainbow indentations to improve code legibility
Increase the font size for better readability

RStudio Settings

Set up RStudio with the file on the left and the console on the right (as the console is more important than the environment tab for most use cases)
Do not save anything into your environment when closing RStudio (Always start with a clean slate to avoid mistakes caused by leftover variables from previous sessions)

Coding Basics

Use Ctrl + Shift + M to create a pipe (|>) without typing special characters
Use Ctrl + Enter to execute separate lines
Use Ctrl + Shift + Enter to execute an entire file
Select a specific part of the code and use Ctrl + Enter to execute only the selected portion (useful for debugging pipe chains step by step)
Open function documentation with F1
Move lines of code with Alt + arrows
Indent with Tab and unindent with Shift + Tab
Multiple cursors with Alt + click
Use Ctrl + 1 and Ctrl + 2 to switch between the console and the current script window (eliminates the need to use the mouse for navigation)

File and Text Search

Use Ctrl + . to search for files AND functions
Use Ctrl + F to search for specific text within the currently open file
Use Ctrl + Shift + F to search for text within all files in the current project

Command Palette and more shortcuts

Use Ctrl + Shift + P to execute commands
Helpful for using shortcuts that you might have forgotten (like expanding all folds or going to a file)
- Run all code chunks (Ctrl + Alt + R)
- Render current document (Ctrl + Shift + K)
- Restarting R Session (Ctrl + Shift + F10)
- Styling code with the styler package
- Running your Golem apps,

Conclusion

Feel free to share your favorite shortcuts, settings, or any other tips I might have missed in the comments below. Have a great day and see you next time. And if you found this helpful, here are some other ways I can help you:

3 Minute Wednesdays: A weekly newsletter with bite-sized tips and tricks for R users
Insightful Data Visualizations for “Uncreative” R Users: A course that teaches you how to leverage {ggplot2} to make charts that communicate effectively without being a design expert.

Albert Rapp

WebDev4R: Shiny Explained

Shiny in a Nutshell

Video

Code

How to Finally Fix Your Tables in Typst PDFs (using {gt} tables)

The Easiest Way to Export Data From Your R-Shiny Apps

Video

Code

Additional Files

File Management With The {fs} Package

Assemble paths

Remove and set extensions

Get directory infos

Iterate over file paths

The difference between = and <- finally explained

Similar assignment

Scoping

Powerup of <-

data.table vs dplyr: A Side-by-Side Comparison

Get data

The general difference

Sort specific columns

Filter for specific rows

Select specific columns

Select many columns by name

Select many columns by type

Compute a new column

Specialty of data.table: modify in place

Calculate multiple new columns

Specialty of data.table: chaining

Calculate summary statistics (without grouping)

Calculate summary statistics (with grouping)

Iterate over multiple columns

5 Levels of Data Wrangling Every R User Must Master

Level 1: Do everything manually

Level 2: Use tidyselect helpers in select()

Level 3: Use across with built-in functions

Level 4: Use across with custom function

Level 5: Use tidyselect helpers in across()

Bonus-Level: Use clean names

Sending e-mails with {blastula}

Creating an email

Filling the email with data

Get email credentials

Sending out the email

Comparing pipes: Base-R |> vs {magrittr} %>%

Keyboard shortcut

Simple function chaining

Form strictness

Standard scenario

Using a placeholder

Using multiple placeholders

Conditional flows

Getting started with SQL with R

Establish a connection

Use database tables

Get data with SQL

dtplyr

Programatically assembling SQL queries with {glue}

Transactions

Creating fancy interactive tables using Internet data with rvest and reactable

Creating interactive tables with reactable

Fake data

Base Layer

Use better column names

Title & Subtitle

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Format numbers

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Add groups

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Add summaries

Hawaiian Pizza Sales in 2015

Based on the fake pizzaplace data from `{gt}`

Make table searchable

Hawaiian Pizza Sales in 2015

Powerup of `<-`

Specialty of `data.table`: modify in place

Specialty of `data.table`: chaining

Level 2: Use tidyselect helpers in `select()`

Level 5: Use tidyselect helpers in `across()`

Programatically assembling SQL queries with `{glue}`

Use `geom_marquee()` instead of `geom_text()`

Add more styles using the `style` aesthetic

Create text boxes using the `width` argument

Count all the things with `count()`

Get the right columns with `select()`

Reduce the rows to specific observations with `filter()`

Calculating and transforming columns with `mutate()`

Reduce your data to key quantities with `summarize()`

Sort your data with `arrange()`