Ambiguity and Confusion

Formats like DD/MM/YYYY and MM/DD/YYYY cause confusion, especially in Orgs. that operate across geographies and multi-national companies (MNC) At a smaller scale, individual gig-workers working with MS Excel files from different clients get confused if the date format is different.

The YYYY-MM-DD format follows a logical order: Year, > Month > Day, from largest to smallest unit. The logic is similar to geographical hierarchy i.e. Earth (Planet) > Asia (Continent) > India (Country) > Karnataka (State) > Bengaluru (City). Largest to Smallest!

This logical structure ensures clarity and consistency in understanding dates across different cultures and languages. For instance:

• January 31, 2024 is represented as 2024-01-31. The sequence is intuitive and unambiguous, reducing the risk of misinterpretation.

• In contrast, formats like MM/DD/YYYY or DD/MM/YYYY can lead to confusion, especially in international communication. For example, 02/03/2024 can mean February 3rd or March 2nd, depending on the chosen format, leading to potential errors.

Sorting and Organization

The YYYY-MM-DD format facilitates sorting dates effortlessly, both alphabetically and chronologically. Whether it’s arranging files on a computer, organizing data in spreadsheets, or storing information in databases, the YYYY-MM-DD format simplifies the process.

• Alphabetical sorting aligns with chronological order, making it easier for computers and humans to process information consistently.

• Other formats may pose challenges in sorting, especially when combining different date formats within the same dataset. Ambiguities arise, leading to errors and confusion during analysis.

Sorting is done based on the Alphabetical order of the characters in the filenames. Let’s evaluate each format on the basis of sorting.

Compatibility and Standardization

YYYY-MM-DD format is widely recognized and accepted in various industries, including technology, finance, and healthcare. Its standardized nature promotes interoperability and data exchange across different systems and platforms.

Many programming languages like and , databases, and software applications prefer the YYYY-MM-DD format for date handling and manipulation. This consistency streamlines development and integration processes. International standards bodies like ISO (International Organization for Standardization) endorse the YYYY-MM-DD format (ISO 8601), further solidifying its status as a global standard.

But, I only work in Microsoft’s Excel!! How does this matter?

Well, you are right!

Spreadsheet applications like MS Excel store Dates as numbers behind the scenes and display them as per the system’s Locale. An Excel file containing Dates (formatted as proper Dates, not dates stored as Text!!! 😱 ) will change the date format accordingly because the underlying number doesn’t change. However, at one glance, it is not possible to guess if a date-like text is a proper Date or Text. Dates stored as text will not change as per system’s Locale and hence may cause errors in calculations and sorting issues. While the YYYY-MM-DD format will resolve sorting issues (due to its inherent alphabetical order) even if stored as text, calculation errors will still occur if dates are stored as text.

Exploring date-related horrors in MS Excel is a topic for a whole other post, which will come soon!

Incidents

While there might not be a single major incident directly caused by unambiguous date formats, the lack of standardized date formats has contributed to numerous misunderstandings, errors, and inefficiencies in various contexts. Here are a few examples:

1. Y2K Bug: One of the most well-known incidents related to date formatting issues is the Y2K (Year 2000) bug. Many computer systems used a two-digit representation for the year, assuming the first two digits were “19” by default (e.g., “98” for 1998). As the year 2000 approached, there was concern that these systems would interpret “00” as 1900 instead of 2000, potentially causing malfunctions or system failures. While not directly caused by unambiguous date formats, the Y2K issue highlighted the importance of standardized and unambiguous date representations in computer systems.

2. Medical Errors: In healthcare settings, incorrect interpretation of dates can lead to medical errors, missed appointments, and treatment delays. For example, confusion between MM/DD/YYYY and DD/MM/YYYY formats could result in misinterpreting appointment dates, prescription refills, or test results, potentially compromising patient care and safety.

3. Financial Transactions: In the financial sector, accurate date representations are crucial for transactions, accounting, and regulatory compliance. Ambiguous date formats can lead to errors in financial reporting, billing, and tax calculations, causing financial discrepancies and regulatory violations.

4. Legal Documents: Ambiguous date formats in legal documents such as contracts, agreements, and court filings can lead to disputes, legal challenges, and contract breaches. Inconsistencies in date interpretation may affect the validity and enforceability of legal agreements, resulting in costly litigation and legal proceedings.

Closing

While these incidents may not be directly attributed to unambiguous date formats, they underscore the importance of standardized date representations in ensuring accuracy, clarity, and interoperability across different systems, industries, and domains. Standardized date formats, such as those defined in ISO 8601, help mitigate the risk of errors, misunderstandings, and disruptions caused by inconsistent date representations, promoting efficiency, reliability, and trust in data exchange and communication.

While it is highly doubtful that this article says anything new compared to already existing literature, the switch to using YYYY-MM-DD format consistently is a small step towards a more inclusive and unambiguous system of time-keeping!

Guess, I woke up and chose violence today!

References

Wikipedia - ISO 8601

Wikipedia - Y2K Bug

Basic Syntax

List comprehensions pack a powerful punch in a compact syntax. They allow you to create lists in a single line, combining iteration and expression evaluation within square brackets. Here’s the most basic structure:

new_list = [expression for item in iterable]

where

• expression is the value or logic applied to each item, which will create the items of the new list
• item is the variable that represents each element of the iterable
• iterable is the variable over which we iterate or ‘loop’. This could be a list, tuple, dictionary, string or anything which can be considered an iterable in Python.

The equivalent for loop for the above operation is as follows:

new_list = []
for item in iterable:
    new_list.append(expression)

Let’s understand this with some working code. Suppose I have a list of numbers and I want a list that contains the same numbers multipled by 2 i.e. doubled.

my_nums = [1, 2, 3, 4, 5]
doubled_nums = [num*2 for num in my_nums]

# Same operation using `for` loop
doubled_nums_for = []
for num in my_nums:
  doubled_nums_for.append(num*2)


print(doubled_nums)
print(doubled_nums_for) # identical to `doubled_nums`

[2, 4, 6, 8, 10]
[2, 4, 6, 8, 10]

In the above Python code, my_nums is the iterable, the num variable in 2nd line represents each item in my_nums and num*2 is the expression or logic that we apply to each num.

let’s take one more example with a dictionary.

my_dict = {'Actor':'Tom', 'Director':'Tony', 'Writer':'Jim'}  
# Bonus for guessing the movie!

# new_list = [expression for item in iterable]
roles = [role for role in my_dict.keys()]
people = [person for person in my_dict.values()]

print(roles)
print(people)

['Actor', 'Director', 'Writer']
['Tom', 'Tony', 'Jim']

As you can see, the item variable can be named anything as this variable is active only within the scope of the list comprehension.

Advanced Syntax: Filtering

We use the following syntax when we want to create a new list with items that satisfy some condition.

filtered_list = [expression for item in iterable if condition]

where

• condition is any logical expression that return True or False

Let’s see an example. Suppose I have a list of sentences and I want to filtered list which has the word ‘whisper’ in them.

sentence_list = [
   "Sunrise paints the clouds in fiery hues, a silent alarm",
   "Raindrops pitter-patter on cobblestones, a playful melo",
   "Ocean waves whisper secrets to the sandy shore, tales o",
   "Owl's amber eyes pierce the moonlit forest, a silent gu",
   "Butterfly wings, stained glass windows fluttering throu",
   "Laughter spills from a cozy cafe window, a warm invitat",
   "Cracked pavement whispers forgotten stories, echoes of ",
   "Starry sky, a canvas splashed with diamonds, whispers o",
   "Spice-laden wind dances through the market, teasing the",
   "Tiny snail embarks on a grand journey, a blade of grass"
]


filtered_list = [sen for sen in sentence_list if 'whisper' in sen]

# Same operation using `for` loop
filtered_list_for = []
for sen in sentence_list:
  if 'whisper' in sen:
    filtered_list_for.append(sen)


print(filtered_list)
print(filtered_list_for) # identical to `filtered_list`

['Ocean waves whisper secrets to the sandy shore, tales o', 'Cracked pavement whispers forgotten stories, echoes of ', 'Starry sky, a canvas splashed with diamonds, whispers o']
['Ocean waves whisper secrets to the sandy shore, tales o', 'Cracked pavement whispers forgotten stories, echoes of ', 'Starry sky, a canvas splashed with diamonds, whispers o']

In the above code, the condition is 'whisper' in sen which returns True or False for every sen sentence.

Let’s look at a more useful example. Here we create a JSON-formatted string using List Comprehension

import pandas as pd
import json

# Sample DataFrame
data = {'name': ['Alice', 'Bob', 'Charlie'], 
        'age': [25, 30, 20], 
        'city': ['New York', 'London', 'Paris']
        }
df = pd.DataFrame(data)

# Convert DataFrame to JSON using list comprehension
json_list = [row.to_json() for index, row in df.iterrows()]

# Convert list to JSON and print
for json_string in json_list:
  print(json.dumps(json.loads(json_string), indent=4))

{
    "name": "Alice",
    "age": 25,
    "city": "New York"
}
{
    "name": "Bob",
    "age": 30,
    "city": "London"
}
{
    "name": "Charlie",
    "age": 20,
    "city": "Paris"
}

Here’s what is happening in the code above.

1. Import Libraries:
   import pandas as pd: Imports the pandas library for working with DataFrames.
   import json: Imports the json library for working with JSON data.

2. Create DataFrame:
   data = {...}: Creates a dictionary containing data for three columns: ‘name’, ‘age’, and ‘city’.
   df = pd.DataFrame(data): Creates a DataFrame df from the dictionary data.

3. Convert DataFrame to JSON List:
   json_list = [row.to_json() for index, row in df.iterrows()]: This line uses list comprehension to convert each row of the DataFrame into a JSON string and stores them in a list called json_list.
iterrows() iterates over the DataFrame, yielding index and row pairs.
   row.to_json() converts each row into a JSON string.

4. Print Pretty-Printed JSON:
   for json_string in json_list:: This loop iterates over each JSON string in the json_list.
print(json.dumps(json.loads(json_string), indent=4)): This line prints the JSON string with proper indentation:
   json.loads(json_string) parses the JSON string into a Python dictionary.
   json.dumps() re-serializes the dictionary back into a JSON string, applying indentation for readability.

Advanced Syntax: If-Else

The If-Else syntax allows us to take one action if the item satisfies a condition and another action if it does not. The syntax is as follows:

new_list = [true_expr if condition else false_expr for item in iterable] 

where

• true_expr is the expression which is evaluated when the item satisfies the condition
• false_expr is the expression which is evaluated when the item does not satisfy the condition

Let’s look at an example of this If-Else syntax. Suppose I have list of numbers with missing values. I want replace the missing values with the average value of the numbers.

import statistics

num_list = [10, 20, None, 40, None, 20, 10]

# Filtering Syntax: Calculate mean with only the numbers which are not None
mean = statistics.mean(num for num in num_list if num is not None)
print(f"{mean=}")

# If-Else Syntax
clean_list = [num if num is not None else mean for num in num_list]

# This can also be written as
clean_list2 = [mean if num is None else num for num in num_list]


# Same operation using `for` loop
clean_list_for = []
for num in num_list:
  if num is None:
    clean_list_for.append(mean)
  else:
    clean_list_for.append(num)

print(clean_list)
print(clean_list2)
print(clean_list_for) # Identical to `clean_list` and `clean_list2`

mean=20
[10, 20, 20, 40, 20, 20, 10]
[10, 20, 20, 40, 20, 20, 10]
[10, 20, 20, 40, 20, 20, 10]

Real-world Usage

I have personally encountered various scenarios in my data journey where I have come across List of Lists! List comprehension is a great way to quickly flatten list of lists in one line of code.

# Create a list of lists containing strings
words = [["hello", "world"], ["how", "are", "you"], ["today"]]

# Nested Syntax
flattened_words = [word for sublist in words for word in sublist]

# Same Operation using `for` loop
flattened_words_for = []
for sublist in words:
  for word in sublist:
    flattened_words_for.append(word)

print(flattened_words)
print(flattened_words_for) # Identical to `flattened_words`

['hello', 'world', 'how', 'are', 'you', 'today']
['hello', 'world', 'how', 'are', 'you', 'today']

So what’s best?

List comprehensions are ideal when:

• Creating a new list based on an existing iterable.
• Applying simple transformations or filtering to elements.
• Prioritizing concise and readable code.

For loops are preferable when:

• Performing complex operations within the loop.
• Needing more control over the iteration process.
• Requiring side effects beyond list creation (e.g., printing, modifying variables).

Conclusion

While list comprehensions offer a concise approach to list creation, for loops remain essential for broader iteration tasks in Python. For new developers, for loops are easier to understand and make far more sense than list comprehensions. They provide greater flexibility and control, allowing for complex operations, multiple statements within each iteration, and handling side effects (like printing, logging) that go beyond mere list creation. However, when the goal is straightforward list generation with simple transformations or filtering, list comprehensions often deliver a more elegant and efficient solution.

References

The long one

Consider the following data table. It has been created from the famous Gapminder dataset. This table shows the average life expectancy in each continent for 2 years. While some of you may say that Gapminder data contains records for a lot more number of years, here we consider just the latest 2 years for ease of explanation and visual purposes. We have added an extra id column for teaching purpose.

my_data is in the long format as we have continent names and year in their own column and average life expectancy values for each unique combination of year and continent. If we want to compare life expectancy across years for each continent, we need to have the life expectancy values for each continent side-by-side for easier viewing i.e. we need to convert to the wide format. To convert this tibble to the wide format, we need to push the year values into the headers and the average_life_expectancy values under the corresponding year column.

The wide one

The wide format of this table would ideally have only continent and columns having each unique value in the year column as a header. In this case, the wide one would look something like the table below.

The wide format has unique values of the column that are not pushed into headers. In this case, the continent column becomes unique for each row.

Let’s recreate the above transformation in R. First, we create the my_data table.

my_data <- data.frame(
  id = 1:10,
  year = c(2002L, 2002L, 2002L, 2002L, 2002L, 2007L, 2007L, 2007L, 2007L, 2007L),
  continent = c("Africa", "Americas", "Asia", "Europe", "Oceania", "Africa", 
                "Americas", "Asia", "Europe", "Oceania"),
  average_life_expectancy = c(53.33, 72.42, 69.23, 76.7, 79.74, 54.81, 73.61, 70.73, 77.65, 80.72)
)

knitr::kable(my_data)

To convert this table into wide format, we use the pivot_wider() function from {tidyr} R package. Let us see how to use this function.

library(tidyr, quietly = TRUE, warn.conflicts = FALSE)

formals(pivot_wider)

$data


$id_cols
NULL

$id_expand
[1] FALSE

$names_from
name

$names_prefix
[1] ""

$names_sep
[1] "_"

$names_glue
NULL

$names_sort
[1] FALSE

$names_vary
[1] "fastest"

$names_expand
[1] FALSE

$names_repair
[1] "check_unique"

$values_from
value

$values_fill
NULL

$values_fn
NULL

$unused_fn
NULL

$...

The result of formals(pivot_wider) tells us that the minimum information needed to use this function is to provide values to the data,names_from and values_from arguments as all other arguments have default values and hence, are optional.

Using only the minimum arguments with pivot_wider(), we get a wide formatted tibble but with missing data!

wide_minimal <- pivot_wider(
                        data        = my_data,
                        names_from  = year,
                        values_from = average_life_expectancy
                        )

knitr::kable(wide_minimal)

So why did NAs appear in the result?

pivot_wider() creates unique combinations of all columns not included in names_from or values_from argument. Therefore, if your dataframe/tibble had a primary key prior to the transformation, the primary key of your transformed “wide” dataframe is your old primary key + unique combinations of all columns not included in names_from or values_from argument. We do have id column as a primary key in the original tibble. This gives an unusable output with NAs for each combination.

To specify which column/s to be made unique, pass their name to the id_cols argument. Here we pass the continent column to the id_cols argument.

my_data_longer <- pivot_wider(
                        data        = my_data,
                        id_cols     = continent, 
                        names_from  = year,
                        values_from = average_life_expectancy
                        )

knitr::kable(my_data_longer)

If you are a visual person like me and wish to see this transformation with explanations, check out this GIF I made using good ol’ PowerPoint.

Conclusion

pivot_wider() is the successor for the great spread() function and has many advantages over the latter. This function has many other arguments that allow some truly great transformations. Mastering this function (and its long counterpart) is a great skill upgrade while massaging your data to make it “tidy”.

Happy Spreading!

References

• Long vs. Wide Data: What’s the Difference?
• Hadley Wickham and Maximilian Girlich (2022). tidyr: Tidy Messy Data. R package version 1.2.0. https://CRAN.R-project.org/package=tidyr
• Yihui Xie (2022). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.39.

The wide one

Consider the following data table. It has been created from the famous Gapminder dataset. This table shows the average life expectancy in each continent for 2 years. While some of you may say that Gapminder data contains records for a lot more number of years, here we consider just the latest 2 years for ease of explanation and visual purposes.

my_data is in the wide format as we have continent names in column headers and average life expectancy values in each of those columns. To convert this tibble to the long format, we need to pull together the continent names in one column and their corresponding values into another column. See Figure 2

The long one

The long format of this table would ideally have only year, continent and average_life_expectancy columns and look something like the table below.

The long format has repeated values of the column that are not gathered/collected. In this case, the year column gets its values repeated for each row.

Let’s recreate the above transformation in R. First, we create the my_data table.

my_data <- data.frame(
  year     = c(2002L, 2007L), 
  Africa   = c(53.33, 54.81), 
  Americas = c(72.42, 73.61), 
  Asia     = c(69.23, 70.73), 
  Europe   = c(76.70, 77.65), 
  Oceania  = c(79.74, 80.72)
)

knitr::kable(my_data)

To convert this table into long format, we use the pivot_longer() function from {tidyr} R package. Let us see how to use this function.

library(tidyr, quietly = TRUE, warn.conflicts = FALSE)

formals(pivot_longer)

$data


$cols


$names_to
[1] "name"

$names_prefix
NULL

$names_sep
NULL

$names_pattern
NULL

$names_ptypes
NULL

$names_transform
NULL

$names_repair
[1] "check_unique"

$values_to
[1] "value"

$values_drop_na
[1] FALSE

$values_ptypes
NULL

$values_transform
NULL

$...

The result of formals(pivot_longer) tells us that the minimum information needed to use this function is to provide values to the data and cols arguments as all other arguments have default values and hence, are optional.

Using only the minimum arguments with pivot_longer(), we get a long formatted tibble with the columns year, name and value.

long_minimal <- pivot_longer(
                        data      = my_data,
                        cols      = c("Africa", "Americas", "Asia", "Europe", "Oceania")
                        )

knitr::kable(long_minimal)

Notice that the continent names and their corresponding average life expectancy values appear in columns named name and value. These are the default column names. We can change these column names by providing our own names to the arguments names_to and values_to.

Since the year column is the only one that remains as is, we can rewrite the above pivot_longer statement as below

my_data_longer <- pivot_longer(data      = my_data,
                               cols      = !year,
                               names_to  = "continent",
                               values_to = "average_life_expectancy")

knitr::kable(my_data_longer)

If you are a visual person like me and wish to see this transformation with explanations, check out this GIF I made using good ol’ Powerpoint.

Conclusion

pivot_longer() is the successor for the great gather() function and has many advantages over the latter. pivot_longer() repeats all the values in the columns that are not included in the cols argument. Therefore, if your dataframe/tibble had a primary key prior to the transformation, the primary key of your transformed “longer” dataframe is your old primary key + the new column created by names_to. This function has many other arguments that allow some truly great transformations. Mastering this function (and its wide counterpart) is a great skill upgrade while massaging your data to make it “tidy”.

Happy Gathering!

References

• Hadley Wickham and Maximilian Girlich (2022). tidyr: Tidy Messy Data. R package version 1.2.0. https://CRAN.R-project.org/package=tidyr
• Yihui Xie (2022). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.39.

What do we mean by Robust Functions?

You must be familiar with the Garbage-In-Garbage-Out philosophy of Software engineering. We can think of it in terms of functions, that, given garbage or bad input, you get garbage or bad output. For a function to be robust, it must behave in a consistent manner for known and correct inputs, however, more importantly, it mustn’t give us garbage for bad inputs. Rather, it must provide useful output (as messages or instructions) which can be further used to inform the end-user about possible problems in the inputs to drive proper usage. The useful output/s in case of bad inputs would ideally be a combination of clean early exit and easy-to-understand error messages. So we shall try to implement Garbage-In-Useful-Info-Out by looking at some ways we can build well-behaved and reliable functions.

Input values passed to a function are more popularly known as arguments or parameters. A robust function must validate the function arguments before proceeding to implement the function logic. If this is not done, then the bad arguments will cause some errors in the logic and display error messages that the end-user may not be familiar with. Worst-case scenario is when the function doesn’t encounter any errors and just gives bad results!! Surely, we do not want this unpredictable behavior.

Our sweet, innocent and naive Function

Consider the following function make_date that takes 3 numeric inputs yyyy, mm and dd and returns a single `Date` object.

make_date <-  function(yyyy, mm, dd) {
  
  # main logic : Concatenate the values and convert to Date
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

my_date <- make_date(yyyy = 2022, mm = 1, dd = 31)
my_date

[1] "2022-01-31"

class(my_date)

[1] "Date"

We will use make_date to demonstrate a couple of scenarios where this function can fail and the methods to safeguard against such scenarios.

Scenario 1: Missing Arguments

The most basic check we should perform before running the function logic is to confirm if all the required arguments are available. Think about how your function should behave if one of the arguments, suppose mm is missing.

make_date(yyyy = 2022, dd = 31)

Error in paste(yyyy, mm, dd, sep = "-"): argument "mm" is missing, with no default

Note that the error message shown to the user, is triggered, not from our function make_date but from the internal paste function. We do not have any control over what error messages are shown when errors occur. In this case, we know specifically that this error is due to a missing argument.

There are two ways to handle missing arguments:

make_date <-  function(yyyy, mm, dd) {
  
  # check missing arguments
  if (missing(yyyy)) stop("argument `yyyy` is required.")
  if (missing(mm))   stop("argument `mm` is required.")
  if (missing(dd))   stop("argument `dd` is required.")
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

# Calling the function without `mm` argument
make_date(yyyy = 2022, dd = 31)

Error in make_date(yyyy = 2022, dd = 31): argument `mm` is required.

dummy_fun <- function(a, b, c) { 
  if(any(missing(a), missing(b), missing(c))) {
    stop("One or more required arguments missing.")
  }
  # Do something...
}
dummy_fun(a = 1)

Error in dummy_fun(a = 1): One or more required arguments missing.

args(paste)

function (..., sep = " ", collapse = NULL, recycle0 = FALSE) 
NULL

make_date <-  function(yyyy, mm = 1, dd = 1) {
  
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

# Calling the function without `mm` and `dd` arguments
make_date(yyyy = 2022) # here, only `yyyy` is the required argument.

Warning in make_date(yyyy = 2022): argument `mm` is missing. Using default value
mm = 1 instead

Warning in make_date(yyyy = 2022): argument `dd` is missing. Using default value
dd = 1 instead

[1] "2022-01-01"

Scenario 2: Invalid Argument Data Type

We have defined make_date to accept 3 numeric arguments i.e. all 3 must be numbers. What would happen if someone tried to call make_date with character, factor or boolean inputs?

make_date(yyyy = "2022", mm = "5", dd = "20") # works!! why?

[1] "2022-05-20"

In this case, the function works because when the arguments are combined into a single string using paste , it matches the format argument of the as.Date function in the main logic of make_date which is as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")

make_date(yyyy = "2022", mm = "May", dd = "1") # works but shows NA !!!

[1] NA

In this case, all the arguments pass the checks but since we pass 2022-May-1 to as.Date which doesn’t match the format = '%Y-%m-%d' thus giving NA.

How do we check if the values provided to the arguments are numbers or number-like? If the values are numbers, we let them pass. But if they are non-numeric, we must check if they can be converted to numbers i.e. we must check if they are number-like. By number-like, I mean, will coercing the value using as.numeric give us a numeric value or NA ? You guessed it right, we will pass the values through as.numeric and check if the output is NA or not.

What are the various data types in R that are not numeric but can look like numbers? We have character, factor and boolean data types which can behave like numbers sometimes. Let’s see a few scenarios.

Year <- c("2022", "TwentyTwo")
Year_num <- as.numeric(Year) # this should show a warning about NAs introduced by coercion

Warning: NAs introduced by coercion

Year_num # must show the number 2022 without quotes and one NA

[1] 2022   NA

Year <- factor(c("2022","2021","TwentyTwo"))
as.numeric(Year)

[1] 2 1 3

YearX <- factor(c("2022", "X"))
as.numeric(YearX)

[1] 1 2

YearY <- factor(2022)
as.numeric(YearY)

[1] 1

make_date <-  function(yyyy, mm = 1, dd = 1) {
  
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  # Check data types
  if (!is.numeric(yyyy) & !is.character(yyyy)) {
    stop("argument `yyyy` must be numeric")
  } else if (is.character(yyyy) & is.na(as.numeric(yyyy))) {
    stop("argument `yyyy` must be numeric")
  }
  if (!is.numeric(mm) & !is.character(mm)) {
    stop("argument `mm` must be numeric")
  } else if (is.character(mm) & is.na(as.numeric(mm))) {
    stop("argument `mm` must be numeric")
  }
  if (!is.numeric(dd) & !is.character(dd)) {
    stop("argument `dd` must be numeric")
  } else if (is.character(dd) & is.na(as.numeric(dd))) {
    stop("argument `dd` must be numeric")
  }
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

# Calling the function with new datatype checks
make_date(yyyy = "2022", mm = "May", dd = "1")

Warning in make_date(yyyy = "2022", mm = "May", dd = "1"): NAs introduced by
coercion

Error in make_date(yyyy = "2022", mm = "May", dd = "1"): argument `mm` must be numeric

make_date(yyyy = "2022", mm = factor("5"), dd = "1")

Error in make_date(yyyy = "2022", mm = factor("5"), dd = "1"): argument `mm` must be numeric

# This function check if value is number or number-like.
is_numberlike <- function(x){
  if (!is.numeric(x) & !is.character(x)) {
    # Early Exit 1 if value is neither numeric nor character
    return(FALSE) 
  } else if (is.character(x) & is.na(as.numeric(x))) {
    # Early Exit 2 if character value is not number-like.
    return(FALSE) 
  }
  return(TRUE)
}

make_date <-  function(yyyy, mm = 1, dd = 1) {
  
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  # Check data types
  if (!is_numberlike(yyyy)) stop("argument `yyyy` must be numeric")
  if (!is_numberlike(mm))   stop("argument `mm` must be numeric")
  if (!is_numberlike(dd))   stop("argument `dd` must be numeric")
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

# Calling the function with new datatype checks
make_date(yyyy = "TwentyTwo", mm = "5", dd = 1)

Warning in is_numberlike(yyyy): NAs introduced by coercion

Error in make_date(yyyy = "TwentyTwo", mm = "5", dd = 1): argument `yyyy` must be numeric

make_date(yyyy = "2022", mm = factor("5"), dd = "1")

Error in make_date(yyyy = "2022", mm = factor("5"), dd = "1"): argument `mm` must be numeric

make_date(yyyy = 2022, mm = 5, dd = "one")

Warning in is_numberlike(dd): NAs introduced by coercion

Error in make_date(yyyy = 2022, mm = 5, dd = "one"): argument `dd` must be numeric

make_date(yyyy = 2022, mm = 1:12, dd = "1")

Warning in if (is.character(x) & is.na(as.numeric(x))) {: the condition has
length > 1 and only the first element will be used

 [1] "2022-01-01" "2022-02-01" "2022-03-01" "2022-04-01" "2022-05-01"
 [6] "2022-06-01" "2022-07-01" "2022-08-01" "2022-09-01" "2022-10-01"
[11] "2022-11-01" "2022-12-01"

mm <- 1:12
paste("Month", mm)

 [1] "Month 1"  "Month 2"  "Month 3"  "Month 4"  "Month 5"  "Month 6" 
 [7] "Month 7"  "Month 8"  "Month 9"  "Month 10" "Month 11" "Month 12"

Scenario 3: Incorrect Argument Size

To ensure make_date gives you just one date, we must ensure that the arguments have just value and is not a vector of multiple values i.e. length(arg)==1. Let’s further add a few checks for the data size of the arguments and rearrange the checks.

make_date <-  function(yyyy, mm = 1, dd = 1) {
  
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  # Check argument lengths
  if (length(yyyy)!=1) stop(paste0("Length of argument `yyyy` is ", length(yyyy),". Must be only 1."))
  if (length(mm)!=1)   stop(paste0("Length of argument `mm` is ", length(mm),". Must be only 1."))
  if (length(dd)!=1)   stop(paste0("Length of argument `dd` is ", length(dd),". Must be only 1."))
  
  # Check data types
  if (!is_numberlike(yyyy)) stop("argument `yyyy` must be numeric")
  if (!is_numberlike(mm))   stop("argument `mm` must be numeric")
  if (!is_numberlike(dd))   stop("argument `dd` must be numeric")
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

# Calling function with new data size checks
make_date(yyyy = 2022, mm = 1:12, dd = "01")

Error in make_date(yyyy = 2022, mm = 1:12, dd = "01"): Length of argument `mm` is 12. Must be only 1.

make_date(yyyy = c("2021","2022"), mm = "1", dd = 1)

Error in make_date(yyyy = c("2021", "2022"), mm = "1", dd = 1): Length of argument `yyyy` is 2. Must be only 1.

make_date(yyyy = 2022, mm = 1, dd = c("1","2"))

Error in make_date(yyyy = 2022, mm = 1, dd = c("1", "2")): Length of argument `dd` is 2. Must be only 1.

A little detour…

So far we checked for missing arguments, arguments with bad data types and arguments with incorrect sizes. We’ve used the stop function along with if to check for all failure conditions and show appropriate error messages. When we use stop, we must specify all the failure conditions and the number of specific error messages goes up as number of arguments increases.

In case of our make_date, if an argument is not missing, it must be a number-like value of length 1. To reduce the number of error messages, we can combine the error messages for data type and length. for eg, the error message could be argument yyyy must be a number-like value of length 1.

Wouldn’t it be easier if we just specify what is the success condition aka the “happy path”, and show error for all other conditions? To do this, we can use the stopifnot function that let’s us specify all the happy paths. See example below.

dummy_sum <- function(a, b, c){
  
  # check missing
  stopifnot(!missing(a) & !missing(b) & !missing(c))
  
  # check argument values
  stopifnot(!is.na(a) & is.numeric(a) & length(a)==1,
            !is.na(b) & is.numeric(b) & length(b)==1,
            !is.na(c) & is.numeric(c) & length(c)==1
            )
  sum(a, b, c)
}

dummy_sum(b = 2, c = 3) # a is missing

Error in dummy_sum(b = 2, c = 3): !missing(a) & !missing(b) & !missing(c) is not TRUE

dummy_sum(a = NA_integer_, b = 2, c = 3) # a has NA value

Error in dummy_sum(a = NA_integer_, b = 2, c = 3): !is.na(a) & is.numeric(a) & length(a) == 1 is not TRUE

dummy_sum(a = 1, b = "2", c = 3) # b has non-numeric value

Error in dummy_sum(a = 1, b = "2", c = 3): !is.na(b) & is.numeric(b) & length(b) == 1 is not TRUE

dummy_sum(a = 1, b = 2, c = 5:7)  # c has length != 1

Error in dummy_sum(a = 1, b = 2, c = 5:7): !is.na(c) & is.numeric(c) & length(c) == 1 are not all TRUE

Note the error messages above. They are not so user-friendly. Luckily, we can specify error messages in stopifnot by providing the error messages as the names of the “happy path” conditions.

dummy_sum <- function(a, b, c){
  
  # check missing
  stopifnot("one or more required arguments missing" = !missing(a) & !missing(b) & !missing(c))
  
  # check argument values
  stopifnot("argument `a` must not be NA, must be a number of length 1" = !is.na(a) & is.numeric(a) & length(a)==1,
            "argument `b` must not be NA, must be a number of length 1" = !is.na(b) & is.numeric(b) & length(b)==1,
            "argument `c` must not be NA, must be a number of length 1" = !is.na(c) & is.numeric(c) & length(c)==1
            )
  sum(a, b, c)
}

dummy_sum(b = 2, c = 3) # a is missing

Error in dummy_sum(b = 2, c = 3): one or more required arguments missing

dummy_sum(a = NA_integer_, b = 2, c = 3) # a has NA value

Error in dummy_sum(a = NA_integer_, b = 2, c = 3): argument `a` must not be NA, must be a number of length 1

dummy_sum(a = 1, b = "2", c = 3) # b has non-numeric value

Error in dummy_sum(a = 1, b = "2", c = 3): argument `b` must not be NA, must be a number of length 1

dummy_sum(a = 1, b = 2, c = 5:7)  # c has length != 1

Error in dummy_sum(a = 1, b = 2, c = 5:7): argument `c` must not be NA, must be a number of length 1

Using stopifnot in our make_date function to combine the datatype and length checks, we get…

make_date <-  function(yyyy, mm = 1, dd = 1) {
  
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  
  # Check argument types and length
  stopifnot(
    "argument `yyyy` must be numeric with length 1" = is_numberlike(yyyy) & length(yyyy)==1,
    "argument `mm` must be numeric with length 1"   = is_numberlike(mm)   & length(mm)==1,
    "argument `dd` must be numeric with length 1"   = is_numberlike(dd)   & length(dd)==1
  )
  
  # main logic
  as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
}

make_date() # no arguments provided

Error in make_date(): argument `yyyy` is required.

make_date(yyyy = 2022, mm = 1:12, dd = 31) # Length mm not equal to 1

Warning in if (is.character(x) & is.na(as.numeric(x))) {: the condition has
length > 1 and only the first element will be used

Error in make_date(yyyy = 2022, mm = 1:12, dd = 31): argument `mm` must be numeric with length 1

make_date(yyyy = 2022, mm = "Jan", dd = 31) # mm is not number-like

Warning in is_numberlike(mm): NAs introduced by coercion

Error in make_date(yyyy = 2022, mm = "Jan", dd = 31): argument `mm` must be numeric with length 1

make_date(yyyy = 2022, dd = 31) # argument mm is missing but should work using default value

Warning in make_date(yyyy = 2022, dd = 31): argument `mm` is missing. Using
default value mm = 1 instead

[1] "2022-01-31"

Scenario 4: Values of Arguments that result in invalid outputs

Finally, what do we do when the arguments provided will definitely give us bad results despite passing all checks? In our case, make_date creates a date but if we give it values that will result in an invalid date, it will give us invalid results (remember Garbage-In-Garbage-Out?).

make_date(yyyy = 2022, mm = 13, dd = 1) # is there a 13th month?

[1] NA

We get NA because as.Date returns NA for invalid inputs with no error messages or warnings! We can check the output and provide a generic error message.

make_date <-  function(yyyy, mm = 1, dd = 1) {
  # check missing arguments
  if (missing(yyyy))  stop("argument `yyyy` is required.") 
  if (missing(mm)) warning("argument `mm` is missing. Using default value mm = 1 instead") 
  if (missing(dd)) warning("argument `dd` is missing. Using default value dd = 1 instead")
  
  
  # Check argument types and length
  stopifnot(
    "argument `yyyy` must be numeric with length 1" = is_numberlike(yyyy) & length(yyyy)==1,
    "argument `mm` must be numeric with length 1"   = is_numberlike(mm)   & length(mm)==1,
    "argument `dd` must be numeric with length 1"   = is_numberlike(dd)   & length(dd)==1
  )
  
  # main logic
  out <- as.Date(paste(yyyy, mm, dd, sep = "-"), format = "%Y-%m-%d")
  if (is.na(out)) {
    stop("Invalid values provided. Please check your inputs.")
  }
  return(out)
}

make_date(yyyy = 2022, mm = 13, dd = 1) # is there a 13th month?

Error in make_date(yyyy = 2022, mm = 13, dd = 1): Invalid values provided. Please check your inputs.

make_date(yyyy = 2022, mm = 2, dd = 31) # are there 31 days in February?

Error in make_date(yyyy = 2022, mm = 2, dd = 31): Invalid values provided. Please check your inputs.

Do you think our function make_date is robust enough?

Conclusion

Making functions robust requires some prior thought about its intended use and audience. Based on this, we can decide what checks to implement, what to skip, whether to stop execution using error messages or to use default values with warnings. Checking for “happy paths” is simpler compared to checking each and every bad input and providing specific error messages. Too many different error messages for the same argument could become a source of frustration of the end user, so consider combining some checks and their error messages to be informative and precise. Robustness, like everything else, in moderation, is good and getting it “just right” takes time and dedicated effort. Happy Coding!

Citations & References

• Techniques for writing robust R programs - LexJansen
• R Programming for Data Science
• A Warning About warning

Quick Recap

In the previous post, we create the .potx template from the .pptx file we wanted to automate and the Excel template with the macro .xlsm that uses the PowerPoint template to create a new .pptx file with given data using VBA.

The report we want to automate is…

…and the Excel and PowerPoint template we created are shown in Figure 2.

In this post, we will write the R script that will first massage the data into desired format and then load the data for one region into the Excel template and execute the VBA macro that will create the PowerPoint file with that data.

Strategy

Before we dive into code, we need to check a few things. We wish to create a presentation for each continent in the Gapminder data. A closer look at the Presentation will tell you what kind of data we need for each slide/graph/table while the Excel template will reveal what should the structure of each dataset should be. While looking into this structure, some questions will pop-up. The idea here is to create the datasets in such a way that they can be easily filtered for each continent and the resultant table can be written to the Excel template without any or very little modification. Let us proceed slide-by-slide.

Creating the datasets

Slide 2

The chart on slide 2 needs raw data structured as below. You will notice that at a continent-level, this table needs a minimum of 5 countries. Do we have any continents in the Gapminder data with less than 5 countries? Yes, we have Oceania with only Australia and New Zealand. For ease of use, let us include these countries along with Asian countries in a new Region variable.

We will create the region variable in the gapminder data. But first, let us load some relevant packages.

Code

options(tidyverse.quiet = TRUE)
library(tidyverse) # duh!
library(rmarkdown) # to display the tables interactively in this post. Not really needed for the final solution.
library(openxlsx) # to write the data to the Excel Template.
# library(RDCOMClient) # to load and run the Excel macro post data load.

Code

# Read in Gapminder data
gp <- gapminder::gapminder

# Create new region variable
gp <- gp %>%
  mutate(region = if_else(as.character(continent) %in% c("Asia","Oceania"),
                          "Asia-Pacific", 
                          as.character(continent)),
         country = as.character(country))

# Keep only relevant columns
gp <- gp %>% select(region, country, year, pop)

# View details
glimpse(gp)

Rows: 1,704
Columns: 4
$ region  <chr> "Asia-Pacific", "Asia-Pacific", "Asia-Pacific", "Asia-Pacific"~
$ country <chr> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanistan", "A~
$ year    <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1997, 20~
$ pop     <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372, 1288~

Now that we have the source data available, we must now create the datasets we need that we can write to the Excel template for each region.

The required table in Figure 3 shows the top 4 countries (based on 2007 population) and all other countries clubbed into ‘others’ in a given region and then the total population of the region on a yearly basis. This table has to be created for all 4 regions.

Code

pop_trend <- gp %>%
  group_by(region, country, year) %>% 
  summarise(pop = sum(pop, na.rm = TRUE),
            .groups = 'drop') %>%
  mutate(pop = round(pop/1E6, 0)) %>% # population in millions
  pivot_wider(names_from = year, values_from = pop, names_sort = TRUE) %>% 
  arrange(desc(`2007`)) # sort by max pop to min pop in latest year i.e. 2007

paged_table(pop_trend)

Now that we have the required columns, let’s plan the row order. We notice that, for each region, we have the top 4 countries (as per 2007) , followed by ‘Others’. Let’s create the top-4 dataset.

Code

top4 <- pop_trend %>% 
  group_by(region) %>% 
  slice_max(`2007`, n = 4, with_ties = FALSE) %>% 
  ungroup()

paged_table(top4)

To create the others dataset, we exclude all countries that are present in the top-4.

Code

others <- pop_trend %>% 
  filter(!country %in% top4$country) %>% 
  group_by(region) %>% 
  summarise(across(.cols = -country, .fns = sum),
            .groups = 'drop') %>% 
  mutate(country = "Others") %>% 
  select(region, country, everything())

paged_table(others)

While we create the top-4 and others datasets separately, we will combine them later at the very last moment before writing them to the Excel template.

Now that we have the datasets needed for 02_chart, let’s proceed to the create 02_table . This table gives you the count of countries that fall under various population ranges.

Let’s create 02_table. To create this table, we first create a new variable called pop_range.

Code

pop_levels <- c('Less than 500K','500K - 1 Million',
                '1M - 10 Million', '10M - 100 Million',
                '100M - 1 Billion', 'More than 1 Billion')

gp2007 <- gp %>% 
  filter(year == 2007) %>% 
  mutate(pop_range = case_when(pop < 5E5 ~ pop_levels[1],
                               pop < 1E6 ~ pop_levels[2],
                               pop < 1E7 ~ pop_levels[3],
                               pop < 1E8 ~ pop_levels[4],
                               pop < 1E9 ~ pop_levels[5],
                               TRUE      ~ pop_levels[6]),
         pop_range = factor(pop_range, levels = pop_levels))

pop_groups <- gp2007 %>% 
  group_by(region, pop_range, .drop = FALSE) %>% 
  summarise(`# of Countries` = n(),
            .groups = 'drop') %>% 
  arrange(region, pop_range) %>% 
  rename(`Population Category` = pop_range)

paged_table(pop_groups)

Slide 3

Slide 3 contains 2 strings and one chart. The data for the chart looks as shown below.

The data for 03_chart is the list of top 10 countries in each region as per latest record i.e. 2007. Let’s create the top10 table.

Code

top10 <- gp %>% 
  filter(year == 2007) %>% 
  group_by(region) %>% 
  slice_max(pop, n = 10, with_ties = FALSE) %>% 
  ungroup() %>% 
  select(-year) %>% 
  mutate(pop = round(pop/1E6, 4)) %>% # population in millions
  set_names(c("region","country","population"))

paged_table(top10)

References & Citations

• Jennifer Bryan (2017). gapminder: Data from Gapminder. R package version 0.3.0. https://CRAN.R-project.org/package=gapminder

• Hadley Wickham, Romain Francois, Lionel Henry and Kirill Muller (2021). dplyr: A Grammar of Data Manipulation. R package version 1.0.7. https://CRAN.R-project.org/package=dplyr

• Hadley Wickham (2021). tidyr: Tidy Messy Data. R package version 1.1.3. https://CRAN.R-project.org/package=tidyr

• Greg Lin (2020). reactable: Interactive Data Tables Based on ‘React Table’. R package version 0.2.3. https://CRAN.R-project.org/package=reactable

• Philipp Schauberger and Alexander Walker (2021). openxlsx: Read, Write and Edit xlsx Files. R package version 4.2.4. https://CRAN.R-project.org/package=openxlsx

• Duncan Temple Lang (NA). RDCOMClient: R-DCOM client. http://www.omegahat.net/RDCOMClient, http://www.omegahat.net http://www.omegahat.net/bugs.

• https://docs.microsoft.com/en-us/office/vba/api/overview/excel

• https://docs.microsoft.com/en-us/office/vba/api/overview/powerpoint

TL;DR

In this 2-part blog, we create a PowerPoint template with named placeholders which we populate from an Excel file using VBA. The Excel file is loaded with data using R with the help of openxlsx package and then the macro is triggered using the RDCOMClient package.

This solution has great potential to give you the same feeling as those Jurassic Park scientists that Dr. Ian Malcolm remarked about!

Advantages of this approach over officer and officedown:

1. Slide/content/header/footer formatting control is in the PowerPoint template rather than R code.
2. All charts are native and can contain any feature (dual axis, mixed data series like bar + line, line + points). All Excel chart-types are available. Go wild!
3. You can use any PowerPoint template design (Yes, even your sad/weird/exciting corporate template!).

Let’s begin.

Suppose we want to automate the following PowerPoint presentation. It contains 3 slides with a title slide and 2 content slides having graphs and tables created from the gapminder dataset. This .pptx file also has a custom footer.

We want to create the same presentation with same structure but at a continent-level. gapminder has data for 5 continents and we wish to create 5 presentations by the end of this.

The PowerPoint Template

In this approach, we start with the PowerPoint presentation file. We will create a template with placeholders and charts with dummy data.

Usually, you would have a copy of the .pptx file you want to automate. Save a copy of it as a PowerPoint Template (.potx), ideally to your R Project¹ folder. In my case, I’ve created a new R Project folder named R2XL2PPT as shown in Figure 2.

Now let us prep the template. If you open the template file by double-click or right-click > New, it would open a fresh .pptx presentation using the template. Right-Click and click Open in the context menu to open the .potx template file for editing.

Once you have the template open, we will add names to all the text placeholders, tables and graphs we wish to update. To update the placeholder name:

1. Select the shape/text-area/table/graph.
2. From Shape Format, click Selection Pane.
3. In the Selection Pane, change the name of the selected item.

Once you set the names of all the items that you want to customise, save the template.

Download the GP_template.potx template here.

The Excel Template

To populate all the named items in the PowerPoint template, we will now create an Excel document which looks identical to the template with respect to the named items. Please see Figure 5.

For every named item, depending on whether it is a textbox or chart or table, we will create a named range for that item. For example, for item 01_title in the PowerPoint template, we create a S1_title named range² (which points to cell C3) as a placeholder for it.

You can set a single Excel cell as named range for each textbox in the PowerPoint template. You can copy-paste tables from Powerpoint to the Excel template directly. The entire table must be set as a named range.

For charts, right-click the chart in PowerPoint and select Edit Data. See Figure 6. An excel worksheet is displayed with the underlying data. Copy-paste the entire data into the Excel template.

For the GP_template.potx, the corresponding excel template XL2PPT.xlsm is shown in Figure 7. Please note that this template does not have the VBA macro yet.

The VBA Macro

We want the VBA macro to:

1. Open a new instance of PowerPoint presentation using the GP_template.potx file.
2. Copy text/numbers from various placeholders and replace existing text/numbers in the PowerPoint presentation.
3. Save the presentation with custom file name with .pptx extension.

The actual mapping of the named ranges in the Excel template to the named shapes in the PowerPoint template happens in the VBA code. However, at this stage, you can actually create the PowerPoint presentation by copying the numbers into the Excel template and hitting the big RUN MACRO button.

For your reference, I am embedding the VBA macro code below. You can download the XL2PPT.xlsm file from here.

R

Suppose we want to create a 10 variations of PowerPoint presentation using the same template. While creating the presentation from Excel is now automated, how about creating the numbers for each of those 10 variations? This is where we bring in R with the openxlsx and RDCOMClient packages. We use the tidyverse set of packages to read in data, clean and massage the data into the various formats we need, openxlsx to write the data (single numbers, text or tables of numbers) to the Excel template and RDCOMClient to run the embedded VBA code in the Excel template.

Check out Part 2 of this blog to see how to run VBA code using R. Let me know if you find this useful or any corrections required in the comments below.

Top-Level data: course

library(dplyr, quietly = TRUE, warn.conflicts = FALSE)
library(rmarkdown, quietly = TRUE, warn.conflicts = FALSE)

course <- tibble(course_id   = 1:4,
                 course_name = paste("Course", LETTERS[1:4]),
                 start_date  = seq.Date(from = lubridate::as_date("2021-01-01"), by = "month", length.out = 4),
                 end_date    = lubridate::ceiling_date(start_date, unit = "month") - 1)

paged_table(course)

Second Level Data: student

set.seed(42)
student <- tibble(s_id      = 1:20,
                  s_name    = paste("Student", LETTERS[1:20]),
                  gender    = sample(c("X","Y","Z"), 20, replace = TRUE),
                  age       = sample(18:35, 20, replace = TRUE),
                  course_id = sample(1:4, 20, replace = TRUE))

paged_table(student)

If we are sourcing data from a database, it is probable that you would see these 2 levels of data in 2 separate tables/views, but most business users are comfortable with MS Excel and want all the data in one sheet!!

So the data actually looks something like this.

combined_df <- left_join(course, student, by = "course_id")

paged_table(combined_df)

Displaying such data in a table causes all the top-level data fields to repeat for every second-level record. You can see that course_id, course_name, start_date and end_date columns repeat for all students who enrolled in the same course. Take a moment to think about how would you display such data in an interactive table in a web page, HTML report or Shiny app.

It is advisable to split such denormalized data into normalized data i.e. create the original top-level and second level tables from the combined_df.

Vanilla reactable

One of my favorite R packages is reactable. The default output creates a neat interactive table with pagination (if data has more than 10 rows) and ability to sort columns.

library(reactable, quietly = TRUE, warn.conflicts = FALSE)

reactable(data = combined_df)

Basic Formatting

With some additional tweaks, we can make it look better.

reactable(
  data       = combined_df,
  compact    = TRUE, # for minimum row height
  filterable = TRUE, # for individual column filters
  striped    = TRUE, # banded rows
  resizable  = TRUE, # for resizable column widths
  columns    = list( # define custom header name, width, alignment etc.
    course_id   = colDef(name = "CID",         width = 50,  align = "center"),
    course_name = colDef(name = "Course Name", width = 140),
    start_date  = colDef(name = "Start Date",  width = 120, align = "center"),
    end_date    = colDef(name = "End Date",    width = 120, align = "center"),
    s_id        = colDef(name = "SID",         width = 70,  align = "center"),
    s_name      = colDef(name = "Student Name"),
    gender      = colDef(name = "Gender",      width = 80,  align = "center"),
    age         = colDef(name = "Age",         width = 50)
  )
)

However, the problem of repeating top-level fields still persists.

Grouping and Aggregating

reactable has a groupBy argument which lets us combined rows with common data fields and the aggregate argument inside colDef lets us define what aggregation to be used for each column of the top-level data.

reactable(
  data       = combined_df,
  compact    = TRUE, # for minimum row height
  filterable = TRUE, # for individual column filters
  striped    = TRUE, # banded rows
  resizable  = TRUE, # for resizable column widths
  groupBy    = "course_id",
  columns    = list(
    # show count of students in each course
    course_id   = colDef(name = "CID",         width = 100,  align = "left",    aggregate = "count"),  
    # show unique course name
    course_name = colDef(name = "Course Name", width = 140,                     aggregate = "unique"), 
    # show unique start date
    start_date  = colDef(name = "Start Date",  width = 120,  align = "center",  aggregate = "unique"), 
    # show unique end date
    end_date    = colDef(name = "End Date",    width = 120,  align = "center",  aggregate = "unique"), 
    s_id        = colDef(name = "SID",         width = 70,   align = "center"),
    s_name      = colDef(name = "Student Name"),
    gender      = colDef(name = "Gender",      width = 80,   align = "center"),
    age         = colDef(name = "Age",         width = 50)
  )
)

In this case, all the columns which are not aggregated remain hidden. Clicking the little triangle in the CID column displays the hidden rows. Looks better, but again, the issue of duplicated data remains.

You can aggregate the second-level columns too, but this distorts the table and frankly, looks ugly. Here I aggregate the SID column in addition to all the other top-level columns.

reactable(
  data       = combined_df,
  compact    = TRUE, # for minimum row height
  filterable = TRUE, # for individual column filters
  striped    = TRUE, # banded rows
  resizable  = TRUE, # for resizable column widths
  groupBy    = "course_id",
  columns    = list(
    course_id   = colDef(name = "CID",         width = 100,  align = "left",    aggregate = "count"),
    course_name = colDef(name = "Course Name", width = 140,                     aggregate = "unique"),
    start_date  = colDef(name = "Start Date",  width = 120,  align = "center",  aggregate = "unique"),
    end_date    = colDef(name = "End Date",    width = 120,  align = "center",  aggregate = "unique"),
    # YIKES!! Aggregating Student ID to show unique ids in each course.
    s_id        = colDef(name = "SID",         width = 70,   align = "center",  aggregate = "unique"), 
    s_name      = colDef(name = "Student Name"),
    gender      = colDef(name = "Gender",      width = 80,   align = "center"),
    age         = colDef(name = "Age",         width = 50)
  )
)

Wouldn’t it be nice if we could display only the top-level columns by default and on clicking the small triangle for a row, show all the second-level columns corresponding to that row only, like a drill-down table?

To do this we need 2 separate tables. Earlier in this post, I said it is advisable to split such denormalized data into normalized data i.e. create the original top-level and second level tables from the combined_df. Let’s recreate the 2 tables.

I want to demonstrate how we go from the combined data to the 2 tables. Hence I will not use the course and student tables created earlier.

Creating the top_level table using just the columns in course. Let’s also create a new column n_students depicting count of students in each course.

top_level <- combined_df %>% 
  # Only course info columns
  count(course_id, course_name, start_date, end_date, name = "n_students") 

paged_table(top_level)

second_level <- combined_df %>% 
  # Only Student info columns with unique identifier for Course
  select(course_id, s_id, s_name, gender, age) %>% 
  arrange(s_id)

paged_table(second_level)

The final Drill-down Table

Now that we have the 2 tables ready, let us now create the final reactable. The trick here is to use the details argument to which we pass another reactable of just the rows with students data corresponding to given course.

reactable(
  data       = top_level,
  compact    = TRUE, # for minimum row height
  filterable = TRUE, # for individual column filters
  striped    = TRUE, # banded rows
  resizable  = TRUE, # for resizable column widths
  columns    = list(
    course_id   = colDef(name = "CID",             width = 50,  align = "center"),
    course_name = colDef(name = "Course Name"), 
    start_date  = colDef(name = "Start Date",      width = 120, align = "center"),
    end_date    = colDef(name = "End Date",        width = 120, align = "center"),
    n_students  = colDef(name = "No. of Students", width = 130, align = "center")
  ),
  details = function(index) { # index is the row number of current row.
    # sub-table of only those students for current row.
    sec_lvl = second_level[second_level$course_id == top_level$course_id[index], ] 
    reactable(data       = sec_lvl,
              compact    = TRUE, 
              filterable = TRUE,
              bordered   = TRUE, 
              resizable  = TRUE,
              columns    = list(
                course_id   = colDef(show = FALSE), # hide the course id column
                s_id        = colDef(name = "SID",    width = 70, align = "center"),
                s_name      = colDef(name = "Student Name"),
                gender      = colDef(name = "Gender", width = 90, align = "center"),
                age         = colDef(name = "Age",    width = 50, align = "center")
              )
              )
  }
)

Since the sub-table is also a reactable, you can go another level down… and another, but please do consider the usability aspect of this feature before taking that decision. I haven’t tried going beyond 2 levels of data myself. Maybe a part 2 to this post??

Conclusion

Drill-down tables let you pack a lot of data in a compact manner and allow use by multiple audiences interested in varying degrees/levels of information. reactable can help create an interactive data table from tabular data with sorting and pagination by default. The data table is an HTML widget that can be used in R Markdown documents and Shiny applications, or viewed from an R console. A lot of features can be enabled/disabled using the basic arguments of the reactable() function and much more using custom JavaScript.

References & Citations

• Greg Lin (2020). reactable: Interactive Data Tables Based on ‘React Table’. R package version 0.2.3. https://CRAN.R-project.org/package=reactable

Inspiration

I regularly deal with event-related information with event date and few other columns like event type, root cause etc. Most reports usually involve calculating number of events that took place on a monthly, quarterly or annual basis, sometimes split by event type, root cause and other columns. After a few reports I realized that I am basically writing the same code over and over again to calculate these KPIs. Keeping the DRY¹ principle in mind, I managed to write a few functions to calculate these KPIs with a few dynamic variables. Following is an attempt to articulate what I learnt while creating those functions.

Data

We shall use the Texas Housing Sales data, available as a tibble in the popular ggplot2 package as reference data. It contains monthly information about the housing market in Texas provided by the TAMU real estate center, https://www.recenter.tamu.edu/. It has 8602 observations and 9 variables.

txhousing <- ggplot2::txhousing
dplyr::glimpse(txhousing)

Rows: 8,602
Columns: 9
$ city      <chr> "Abilene", "Abilene", "Abilene", "Abilene", "Abilene", "Abil~
$ year      <int> 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, ~
$ month     <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, ~
$ sales     <dbl> 72, 98, 130, 98, 141, 156, 152, 131, 104, 101, 100, 92, 75, ~
$ volume    <dbl> 5380000, 6505000, 9285000, 9730000, 10590000, 13910000, 1263~
$ median    <dbl> 71400, 58700, 58100, 68600, 67300, 66900, 73500, 75000, 6450~
$ listings  <dbl> 701, 746, 784, 785, 794, 780, 742, 765, 771, 764, 721, 658, ~
$ inventory <dbl> 6.3, 6.6, 6.8, 6.9, 6.8, 6.6, 6.2, 6.4, 6.5, 6.6, 6.2, 5.7, ~
$ date      <dbl> 2000.000, 2000.083, 2000.167, 2000.250, 2000.333, 2000.417, ~

We shall refer the above data in all the following sections.

select()

When using dplyr functions, the two most popular ways to pass column names is either as bare names i.e. column names without enclosing them in quotes like sales or volume OR pass them as a character string like double-quote “sales” or single-quote ‘volume’. You could also pass a character vector like c("sales", "volume"). In this section we will explore the 3 ways to dynamically select the columns we want.

select_raw <- function(df, var) {
  
  # embrace of curly-curly {{}} brackets
  dplyr::select(.data = df, {{var}}) %>%     
    head()
}

# pass single raw name
select_raw(txhousing, sales)

# A tibble: 6 x 1
  sales
  <dbl>
1    72
2    98
3   130
4    98
5   141
6   156

# pass a vector of raw names for multiple columns
select_raw(txhousing, c(sales, volume))      

# A tibble: 6 x 2
  sales   volume
  <dbl>    <dbl>
1    72  5380000
2    98  6505000
3   130  9285000
4    98  9730000
5   141 10590000
6   156 13910000

my_select <- function(df, ...) {
  dplyr::select(.data = df, ...) %>% 
    head()
}

# pass multiple raw names directly
my_select(txhousing, sales, volume)          

# A tibble: 6 x 2
  sales   volume
  <dbl>    <dbl>
1    72  5380000
2    98  6505000
3   130  9285000
4    98  9730000
5   141 10590000
6   156 13910000

my_select_char <- function(df, cols) {
  dplyr::select(.data = df, dplyr::all_of(cols)) %>% 
    head()
}

my_cols <- c("sales","volume")
my_select_char(txhousing, my_cols)

# A tibble: 6 x 2
  sales   volume
  <dbl>    <dbl>
1    72  5380000
2    98  6505000
3   130  9285000
4    98  9730000
5   141 10590000
6   156 13910000

filter()

In the previous section, we passed column names either as bare names or character strings. filter() takes one or more expressions/conditions that result in a logical vector, with same length as number of rows in the data.frame/tibble and returns only those rows for which the expression/condition returns TRUE. Following are 2 ways to pass these logical expressions/conditions. I’m using expression and condition interchangeably here. In this context, a condition is an expression that results in a boolean TRUE/FALSE result.

filter_raw <- function(df, cond) {
  
  # embrace of curly-curly {{}} brackets
  dplyr::filter(.data = df, {{cond}})        
}

# Pass a single raw criterion
filter_raw(txhousing, sales > 8000)

# A tibble: 10 x 9
   city     year month sales     volume median listings inventory  date
   <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
 1 Houston  2006     5  8040 1602621368 151200    35398       5.5 2006.
 2 Houston  2006     6  8628 1795898108 155200    36281       5.6 2006.
 3 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
 4 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
 5 Houston  2013     8  8155 2083377894 186700    21366       3.3 2014.
 6 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.
 7 Houston  2014     7  8391 2278932511 199700    20214       3   2014.
 8 Houston  2014     8  8167 2195184825 202400    20007       2.9 2015.
 9 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
10 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.

filter_raw(txhousing, c(sales > 8000, year > 2010))

Error in `dplyr::filter()`:
! Problem while computing `..1 = c(sales > 8000, year > 2010)`.
x Input `..1` must be of size 8602 or 1, not size 17204.

A <- c(TRUE, TRUE)      # boolean vector of length = 2
B <- c(FALSE, FALSE)    # boolean vector of length = 2
X <- c(A, B)
X

[1]  TRUE  TRUE FALSE FALSE

my_filter <- function(df, ...) { 
  
  # pass the dots argument
  dplyr::filter(.data = df, ...)
  }

# pass multiple raw criteria
my_filter(txhousing, sales > 8000, year > 2010) 

# A tibble: 8 x 9
  city     year month sales     volume median listings inventory  date
  <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
2 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
3 Houston  2013     8  8155 2083377894 186700    21366       3.3 2014.
4 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.
5 Houston  2014     7  8391 2278932511 199700    20214       3   2014.
6 Houston  2014     8  8167 2195184825 202400    20007       2.9 2015.
7 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
8 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.

dplyr::filter(txhousing, "sales > 8000")

Error in `dplyr::filter()`:
! Problem while computing `..1 = "sales > 8000"`.
x Input `..1` must be a logical vector, not a character.

my_filter_string <- function(df, cond) {
  
  # convert text to raw criterion
  dplyr::filter(.data = df, eval(parse(text = cond))) 
}

# pass single text string as criteria
my_filter_string(txhousing, "sales > 8000")  

# A tibble: 10 x 9
   city     year month sales     volume median listings inventory  date
   <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
 1 Houston  2006     5  8040 1602621368 151200    35398       5.5 2006.
 2 Houston  2006     6  8628 1795898108 155200    36281       5.6 2006.
 3 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
 4 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
 5 Houston  2013     8  8155 2083377894 186700    21366       3.3 2014.
 6 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.
 7 Houston  2014     7  8391 2278932511 199700    20214       3   2014.
 8 Houston  2014     8  8167 2195184825 202400    20007       2.9 2015.
 9 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
10 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.

my_filter_strings <- function(df, cond) { 
  
  # combine all criteria
  filter_text <- paste0("(", cond, ")", collapse = " & ")
  
  # (OPTIONAL) show the combined filter string
  message("Filter Condition: ", filter_text)
  
  # convert text to raw criterion
  dplyr::filter(.data = df, eval(parse(text = filter_text)))
  }

my_filter_criteria <- c("sales > 8000", "year > 2010")
my_filter_strings(txhousing, my_filter_criteria)

Filter Condition: (sales > 8000) & (year > 2010)

# A tibble: 8 x 9
  city     year month sales     volume median listings inventory  date
  <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
2 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
3 Houston  2013     8  8155 2083377894 186700    21366       3.3 2014.
4 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.
5 Houston  2014     7  8391 2278932511 199700    20214       3   2014.
6 Houston  2014     8  8167 2195184825 202400    20007       2.9 2015.
7 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
8 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.

my_filter_criteria_with_OR <- c("sales > 8000 | sales < 50", "year > 2010")
my_filter_strings(txhousing, my_filter_criteria_with_OR)

Filter Condition: (sales > 8000 | sales < 50) & (year > 2010)

# A tibble: 315 x 9
   city         year month sales  volume median listings inventory  date
   <chr>       <int> <int> <dbl>   <dbl>  <dbl>    <dbl>     <dbl> <dbl>
 1 Brownsville  2011     1    48 4974408  83300      784      12.6 2011 
 2 Brownsville  2011     2    47 5558575 101400      776      12.7 2011.
 3 Brownsville  2011     7    47 4807019  91200      749      13.1 2012.
 4 Brownsville  2011    12    39 4203440  86800      726      12.4 2012.
 5 Brownsville  2012     1    43 3892348  85000      791      13.6 2012 
 6 Brownsville  2012     3    27 2976148  93800      734      13.3 2012.
 7 Brownsville  2012    11    41 5115393  99000      807      14   2013.
 8 Brownsville  2013    11    38 4824930 108000      859      13.4 2014.
 9 Brownsville  2015     1    41 5400796  97000      733      10.7 2015 
10 Galveston    2011     1    43 8882961 170000     1015      13.7 2011 
# ... with 305 more rows

mutate()

mutate() allows you to add new columns or modify existing columns. In the example below, we will create a new column volume_in_millions from the existing column volume. The names of both the columns can be passed to the function either as raw names or character strings.

mutate_raw <- function(df, new_col_raw, old_col_raw, num = 1) { 
  dplyr::mutate(.data = df, {{new_col_raw}} := {{old_col_raw}}/num) %>% 
    head()
}

txhousing %>% 
  select(city, year, month, volume) %>% 
  # pass raw column names w/o quotes
  mutate_raw(vol_in_millions, volume, 1E6) 

# A tibble: 6 x 5
  city     year month   volume vol_in_millions
  <chr>   <int> <int>    <dbl>           <dbl>
1 Abilene  2000     1  5380000            5.38
2 Abilene  2000     2  6505000            6.50
3 Abilene  2000     3  9285000            9.28
4 Abilene  2000     4  9730000            9.73
5 Abilene  2000     5 10590000           10.6 
6 Abilene  2000     6 13910000           13.9 

mutate_text <- function(df, new_col_str, old_col_str, num = 1) { 
  dplyr::mutate(.data = df, {{new_col_str}} := df[[old_col_str]]/num) %>% 
    head()
}

txhousing %>% 
  select(city, year, month, volume) %>%
  # pass column names as strings
  mutate_text("vol_in_millions", "volume", 1E6) 

# A tibble: 6 x 5
  city     year month   volume vol_in_millions
  <chr>   <int> <int>    <dbl>           <dbl>
1 Abilene  2000     1  5380000            5.38
2 Abilene  2000     2  6505000            6.50
3 Abilene  2000     3  9285000            9.28
4 Abilene  2000     4  9730000            9.73
5 Abilene  2000     5 10590000           10.6 
6 Abilene  2000     6 13910000           13.9 

mutate_var <- function(df, new_col_var, old_col_var, num = 1) {
  dplyr::mutate(.data = df, "{new_col_var}" := df[[old_col_var]]/num) %>% 
    head()
}

new_var <- "vol_in_millions"
old_var <- "volume"

txhousing %>% 
  select(city, year, month, volume) %>%
  # pass column names as variables
  mutate_var(new_var, old_var, 1E6)  

# A tibble: 6 x 5
  city     year month   volume vol_in_millions
  <chr>   <int> <int>    <dbl>           <dbl>
1 Abilene  2000     1  5380000            5.38
2 Abilene  2000     2  6505000            6.50
3 Abilene  2000     3  9285000            9.28
4 Abilene  2000     4  9730000            9.73
5 Abilene  2000     5 10590000           10.6 
6 Abilene  2000     6 13910000           13.9 

arrange()

arrange() sorts the rows of a data frame by the values of selected columns. By default, it sorts in Ascending order. To force a column to sort in Descending order, we must use the desc() function.

arrange_raw <- function(df, var) {
  
  # embrace of curly-curly {{}} brackets
  dplyr::arrange(.data = df, {{var}}) %>%    
    head()
}

arrange_raw(txhousing, sales)

# A tibble: 6 x 9
  city                year month sales  volume median listings inventory  date
  <chr>              <int> <int> <dbl>   <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 San Marcos          2011    10     6 1156999 180000      163       8.3 2012.
2 Harlingen           2000     7     9 1110000  87500      719      30.8 2000.
3 South Padre Island  2011     1     9 2088500 225000     1258      55.7 2011 
4 San Marcos          2011     1    10 1482310 140000      165       7.5 2011 
5 San Marcos          2011    12    10 1561250 140000      148       8   2012.
6 San Marcos          2014    11    10 1506878 146700       96       4   2015.

arrange_raw(txhousing, desc(sales))

# A tibble: 6 x 9
  city     year month sales     volume median listings inventory  date
  <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.
2 Houston  2006     6  8628 1795898108 155200    36281       5.6 2006.
3 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
4 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
5 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
6 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.

arrange_raw(txhousing, c(sales, volume))

Error in `dplyr::arrange()`:
! Problem with the implicit `transmute()` step.
x Problem while computing `..1 = c(sales, volume)`.
x `..1` must be size 8602 or 1, not 17204.

arrange_raw_multiple <- function(df, ...) {
  dplyr::arrange(.data = df, ...) %>% 
    head()
}

arrange_raw_multiple(txhousing, city, sales)

# A tibble: 6 x 9
  city     year month sales  volume median listings inventory  date
  <chr>   <int> <int> <dbl>   <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Abilene  2003     1    68 5385000  70000      668       5.4  2003
2 Abilene  2011     1    68 8834493 123300      809       6.1  2011
3 Abilene  2009     1    70 8414801  92900      861       6.3  2009
4 Abilene  2000     1    72 5380000  71400      701       6.3  2000
5 Abilene  2010     1    73 9130783 112200      868       6.4  2010
6 Abilene  2001     1    75 5730000  64500      779       6.8  2001

arrange_raw_multiple(txhousing, city, desc(sales))

# A tibble: 6 x 9
  city     year month sales   volume median listings inventory  date
  <chr>   <int> <int> <dbl>    <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Abilene  2015     7   268 45845730 148700      986       5   2016.
2 Abilene  2015     6   260 41396230 141500      965       5   2015.
3 Abilene  2007     7   239 29315000 114300      940       5.2 2008.
4 Abilene  2013     8   236 30777727 120000      976       5.4 2014.
5 Abilene  2014     7   231 35861350 145800     1033       5.8 2014.
6 Abilene  2005     6   230 24050000  92500      664       4.1 2005.

arrange_str <- function(df, var, .desc = FALSE) {
  if (.desc) {
    dplyr::arrange(.data = df, desc(df[[var]])) %>% head()
  } else {
    dplyr::arrange(.data = df, df[[var]]) %>% head()
  }
}

arrange_str(txhousing, "sales")

# A tibble: 6 x 9
  city                year month sales  volume median listings inventory  date
  <chr>              <int> <int> <dbl>   <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 San Marcos          2011    10     6 1156999 180000      163       8.3 2012.
2 Harlingen           2000     7     9 1110000  87500      719      30.8 2000.
3 South Padre Island  2011     1     9 2088500 225000     1258      55.7 2011 
4 San Marcos          2011     1    10 1482310 140000      165       7.5 2011 
5 San Marcos          2011    12    10 1561250 140000      148       8   2012.
6 San Marcos          2014    11    10 1506878 146700       96       4   2015.

arrange_str(txhousing, "sales", .desc = TRUE)

# A tibble: 6 x 9
  city     year month sales     volume median listings inventory  date
  <chr>   <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Houston  2015     7  8945 2568156780 217600    23875       3.4 2016.
2 Houston  2006     6  8628 1795898108 155200    36281       5.6 2006.
3 Houston  2013     7  8468 2168720825 187800    21497       3.3 2014.
4 Houston  2015     6  8449 2490238594 222400    22311       3.2 2015.
5 Houston  2013     5  8439 2121508529 186100    20526       3.3 2013.
6 Houston  2014     6  8391 2342443127 211200    19725       2.9 2014.

arrange_str_multiple <- function(df, var, desc = FALSE) {
  if (desc) {
    dplyr::arrange(.data = df, desc(df[var])) %>% head()
  } else {
    dplyr::arrange(.data = df, df[var]) %>% head()
  }
}

# This function arranges the dataframe either all ascending
# or all descending. Definitely need a better example.

arrange_str_multiple(txhousing, c("year", "month", "sales"))

# A tibble: 6 x 9
  city         year month sales  volume median listings inventory  date
  <chr>       <int> <int> <dbl>   <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Paris        2000     1    19 1440000  71700      286       7.5  2000
2 San Marcos   2000     1    22 2380000 106700      190       6.3  2000
3 Lufkin       2000     1    28 2280000  68000       NA      NA    2000
4 Harlingen    2000     1    31 3910000  87500      644      24.9  2000
5 Galveston    2000     1    37 4555000  95000      636       9.1  2000
6 Port Arthur  2000     1    40 3090000  68300      314       5.6  2000

arrange_str_multiple(txhousing, c("year", "month", "sales"), desc = TRUE)

# A tibble: 6 x 9
  city           year month sales     volume median listings inventory  date
  <chr>         <int> <int> <dbl>      <dbl>  <dbl>    <dbl>     <dbl> <dbl>
1 Houston        2015     7  8945 2568156780 217600    23875       3.4 2016.
2 Dallas         2015     7  7038 2021907410 233000    12292       2.4 2016.
3 Austin         2015     7  3466 1150381553 264600     7913       3   2016.
4 San Antonio    2015     7  2962  704891602 198100     9462       4.1 2016.
5 Collin County  2015     7  1861  613669702 292600     2809       2.1 2016.
6 Fort Bend      2015     7  1372  431875327 280400     3328       3.1 2016.

group_by()

In group_by(), we select which columns to, well, group by! (Damn these well-named functions!). So one can use the same techniques as select() to choose the columns.

In the following examples, we will create only one summarised value total_sales for simplicity.

group_raw <- function(df, grp) {
  df %>% 
    group_by({{grp}}) %>% 
    summarise(total_sales = sum(sales, na.rm = TRUE),
              .groups = 'drop')  %>% 
    head(n=5)
}

# Sum of sales per year
group_raw(txhousing, year)

# A tibble: 5 x 2
   year total_sales
  <int>       <dbl>
1  2000      222483
2  2001      231453
3  2002      234600
4  2003      253909
5  2004      283999

# Sum of sales per month
group_raw(txhousing, month)       

# A tibble: 5 x 2
  month total_sales
  <int>       <dbl>
1     1      245924
2     2      296410
3     3      386909
4     4      397332
5     5      448968

group_raw_multiple <- function(df, ...) {
  df %>% 
    group_by(...) %>% 
    summarise(total_sales = sum(sales, na.rm = TRUE),
              .groups = 'drop')  %>% 
    head(n = 5)
}

# Sum of sales per year
group_raw_multiple(txhousing, year)

# A tibble: 5 x 2
   year total_sales
  <int>       <dbl>
1  2000      222483
2  2001      231453
3  2002      234600
4  2003      253909
5  2004      283999

# Sum of sales per month
group_raw_multiple(txhousing, year, month)     

# A tibble: 5 x 3
   year month total_sales
  <int> <int>       <dbl>
1  2000     1       11411
2  2000     2       15674
3  2000     3       20202
4  2000     4       18658
5  2000     5       22388

group_str <- function(df, grp) {
  df %>% 
    group_by(df[grp]) %>% 
    summarise(total_sales = sum(sales, na.rm = TRUE),
              .groups = 'drop')  %>% 
    head(n=5)
}

# Sum of sales per year
group_str(txhousing, "year")

# A tibble: 5 x 2
   year total_sales
  <int>       <dbl>
1  2000      222483
2  2001      231453
3  2002      234600
4  2003      253909
5  2004      283999

# Sum of sales per month
group_str(txhousing, c("year", "month"))       

# A tibble: 5 x 3
   year month total_sales
  <int> <int>       <dbl>
1  2000     1       11411
2  2000     2       15674
3  2000     3       20202
4  2000     4       18658
5  2000     5       22388

# The same column names can be passed as variables containing the character names
yr <- "year"
group_str(txhousing, yr)

# A tibble: 5 x 2
   year total_sales
  <int>       <dbl>
1  2000      222483
2  2001      231453
3  2002      234600
4  2003      253909
5  2004      283999

yrmon <- c("year", "month")
group_str(txhousing, yrmon)

# A tibble: 5 x 3
   year month total_sales
  <int> <int>       <dbl>
1  2000     1       11411
2  2000     2       15674
3  2000     3       20202
4  2000     4       18658
5  2000     5       22388

group_raw2 <- function(df, grp, sumvar) {
  df %>% 
    group_by({{grp}}) %>% 
    summarise("total_{{sumvar}}" := sum({{sumvar}}, na.rm = TRUE),
              .groups = 'drop')  %>% 
    head(n=5)
}

# Sum of sales per year
group_raw2(txhousing, year, sales)

# A tibble: 5 x 2
   year total_sales
  <int>       <dbl>
1  2000      222483
2  2001      231453
3  2002      234600
4  2003      253909
5  2004      283999

# Sum of listings per month
group_raw2(txhousing, month, listings)

# A tibble: 5 x 2
  month total_listings
  <int>          <dbl>
1     1        1854661
2     2        1888104
3     3        1949187
4     4        1991278
5     5        2038932

(Slightly Better) Examples

The txhousing is a city-wise monthly sales and volume dataset. It has a year and month column. Let us create a date column and keep only those columns relevant for our custom tx_summary() function.

small_df <- txhousing %>% 
  mutate(date = lubridate::as_date(glue::glue("{year}-{month}-01"))) %>% 
  select(city, date, sales, volume)

create_ymq <- function(df, date_col) {
  stopifnot(inherits(df, "data.frame"))
  stopifnot(class(df %>% dplyr::pull({{date_col}})) == 'Date')
  mutate(df,
         Year = lubridate::year({{date_col}}),
         nHalf = lubridate::semester({{date_col}}),
         yHalf = lubridate::semester({{date_col}}, with_year = TRUE),
         dHalf = paste0(lubridate::semester({{date_col}}), "H", format({{date_col}},"%y")),
         nQtr = lubridate::quarter({{date_col}}),
         yQtr = lubridate::quarter({{date_col}}, with_year = TRUE),
         dQtr = paste0(lubridate::quarter({{date_col}}),"Q", format({{date_col}},"%y")),
         Month = lubridate::month({{date_col}}),
         yMonth = as.numeric(format({{date_col}}, "%Y.%m")),
         dMonth = format({{date_col}}, "%b %Y")
         )
}

create_ymq(df = small_df, date_col = date) %>% glimpse()

Rows: 8,602
Columns: 14
$ city   <chr> "Abilene", "Abilene", "Abilene", "Abilene", "Abilene", "Abilene~
$ date   <date> 2000-01-01, 2000-02-01, 2000-03-01, 2000-04-01, 2000-05-01, 20~
$ sales  <dbl> 72, 98, 130, 98, 141, 156, 152, 131, 104, 101, 100, 92, 75, 112~
$ volume <dbl> 5380000, 6505000, 9285000, 9730000, 10590000, 13910000, 1263500~
$ Year   <dbl> 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 200~
$ nHalf  <int> 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, ~
$ yHalf  <dbl> 2000.1, 2000.1, 2000.1, 2000.1, 2000.1, 2000.1, 2000.2, 2000.2,~
$ dHalf  <chr> "1H00", "1H00", "1H00", "1H00", "1H00", "1H00", "2H00", "2H00",~
$ nQtr   <int> 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 1, 1, 1, 2, 2, 2, 3, 3, 3, ~
$ yQtr   <dbl> 2000.1, 2000.1, 2000.1, 2000.2, 2000.2, 2000.2, 2000.3, 2000.3,~
$ dQtr   <chr> "1Q00", "1Q00", "1Q00", "2Q00", "2Q00", "2Q00", "3Q00", "3Q00",~
$ Month  <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, ~
$ yMonth <dbl> 2000.01, 2000.02, 2000.03, 2000.04, 2000.05, 2000.06, 2000.07, ~
$ dMonth <chr> "Jan 2000", "Feb 2000", "Mar 2000", "Apr 2000", "May 2000", "Ju~

tx_summary <- function(df, grp_col, sum_col) {
  df %>% 
    group_by(city, {{grp_col}}) %>% 
    summarise("total_{{sum_col}}" := sum({{sum_col}}, na.rm = TRUE), .groups = 'drop')
}

small_df_with_date_cols <- small_df %>% 
  create_ymq(date_col = date)

# Annual Sales per city
small_df_with_date_cols %>% 
  tx_summary(grp_col = Year, sum_col = sales)

# A tibble: 736 x 3
   city     Year total_sales
   <chr>   <dbl>       <dbl>
 1 Abilene  2000        1375
 2 Abilene  2001        1431
 3 Abilene  2002        1516
 4 Abilene  2003        1632
 5 Abilene  2004        1830
 6 Abilene  2005        1977
 7 Abilene  2006        1997
 8 Abilene  2007        2003
 9 Abilene  2008        1651
10 Abilene  2009        1634
# ... with 726 more rows

# Half Yearly volumes per city
small_df_with_date_cols %>% 
  tx_summary(grp_col = yHalf, sum_col = volume)

# A tibble: 1,472 x 3
   city    yHalf total_volume
   <chr>   <dbl>        <dbl>
 1 Abilene 2000.     55400000
 2 Abilene 2000.     53175000
 3 Abilene 2001.     55795000
 4 Abilene 2001.     58570000
 5 Abilene 2002.     55305000
 6 Abilene 2002.     63370000
 7 Abilene 2003.     58175000
 8 Abilene 2003.     77500000
 9 Abilene 2004.     74205000
10 Abilene 2004.     85465000
# ... with 1,462 more rows

# Quarterly Sales per city
small_df_with_date_cols %>% 
  tx_summary(grp_col = yQtr, sum_col = sales)

# A tibble: 2,898 x 3
   city     yQtr total_sales
   <chr>   <dbl>       <dbl>
 1 Abilene 2000.         300
 2 Abilene 2000.         395
 3 Abilene 2000.         387
 4 Abilene 2000.         293
 5 Abilene 2001.         305
 6 Abilene 2001.         394
 7 Abilene 2001.         401
 8 Abilene 2001.         331
 9 Abilene 2002.         295
10 Abilene 2002.         425
# ... with 2,888 more rows

# Monthly Volumes per city
small_df_with_date_cols %>% 
  tx_summary(grp_col = yMonth, sum_col = volume)

# A tibble: 8,602 x 3
   city    yMonth total_volume
   <chr>    <dbl>        <dbl>
 1 Abilene  2000.      5380000
 2 Abilene  2000.      6505000
 3 Abilene  2000.      9285000
 4 Abilene  2000.      9730000
 5 Abilene  2000.     10590000
 6 Abilene  2000.     13910000
 7 Abilene  2000.     12635000
 8 Abilene  2000.     10710000
 9 Abilene  2000.      7615000
10 Abilene  2000.      7040000
# ... with 8,592 more rows

Conclusion

The ability to create such dynamic functions, enabled by the wonderful {dplyr} package, allows us to level-up in terms of programming with R and helps make our code neat and tidy.

References

• Hadley Wickham, Romain François, Lionel Henry and Kirill Müller (2022). dplyr: A Grammar of Data Manipulation. R package version 1.0.9. https://CRAN.R-project.org/package=dplyr
• https://dplyr.tidyverse.org/articles/programming.html
• H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016. https://ggplot2.tidyverse.org
• Jim Hester and Jennifer Bryan (2022). glue: Interpreted String Literals. R package version 1.6.2. https://CRAN.R-project.org/package=glue