Locus: Richard Careaga

Composition

Thu, 27 Aug 2020 00:00:00 +0000

Just as visualizing tensors degrades as n > 3, juggling pieces in an R script can get jiggly when they do not %>% neatly. A post recently reminded me so.

Despite strides in deconstructing R to make it more commercially attractive no obvious point (aside from the richness of statistical subroutines) in using it appears if its functional nature is neglected.

If conceptualized as school algebra (fx)=y writ large, every discrete R tasks is to find or compose f.

# https://community.rstudio.com/t/appending-dataframes-to-list-named-by-variable/77689/5
suppressPackageStartupMessages({library(dplyr)
                                library(purrr)
                                library(tidyr)
                              })
f <- function(x) {
  map(1:length(mk_vs()),sub_df) %>% set_names(.,mk_names(mk_vs(),x))
}

mk_synth_df <- function() {
  set.seed(137)
  ID <- sample(1:100, 50, replace=TRUE)
  N <- sample(1:100, 50, replace = TRUE)
  ratio <- rnorm(50, mean = 0.5, sd=0.1)
  data.frame(ID,N,ratio)
}

mk_list <- function() {
  set.seed(137)
  list("item1" = sample(1:100, 100), "item2" = sample(1:100, 100))
}

mk_vs <- function() {
  set.seed(137)
  v1 <- sample(1:100, 10, replace = TRUE)
  v2 <- sample(1:100, 10, replace = TRUE)
  v3 <- sample(1:100, 10, replace = TRUE)
  list("res1" = v1, "res2" = v2, "res3" = v3)
}

mk_names <- function(x,y) {
  paste(names(x)[1:length(x)],y, sep="")
}

sub_df <- function(x) {
  drop_na(mk_synth_df()[mk_vs()[[x]],])
}

f("_dFList")
#> $res1_dFList
#>    ID  N     ratio
#> 34 47 27 0.4427454
#> 8  96 22 0.5997265
#> 38 71 44 0.6615295
#> 39 14 83 0.4865537
#> 35 15 59 0.4820364
#> 
#> $res2_dFList
#>    ID  N     ratio
#> 30 35 89 0.6159864
#> 48 38 43 0.3970454
#> 13 13 21 0.4866509
#> 43 51 14 0.4953379
#> 14 43 55 0.5479219
#> 22 48 96 0.5725216
#> 4  38 44 0.4549781
#> 6  39 25 0.7056911
#> 
#> $res3_dFList
#>      ID  N     ratio
#> 15   14 89 0.5237554
#> 48   38 43 0.3970454
#> 32   86 89 0.5995506
#> 13   13 21 0.4866509
#> 14   43 55 0.5479219
#> 35   15 59 0.4820364
#> 35.1 15 59 0.4820364

^{Created on 2020-08-27 by the reprex package (v0.3.0)}

Category theory, first steps

Tue, 25 Aug 2020 00:00:00 +0000

observation is “lossy” – Fong & Spivak¹

compositionality systems or relationships combined to form new systems or relationships

structures and structure-preserving maps $f: X \rightarrow Y$: which aspects of $X$ are preserved by $f$?

Consider functions in $\mathbb{R}$: only some preserve order, distance, sum:

A function $f: \mathbb{R} \Rightarrow \mathbb{R}$ is said to be

order-preserving $\iff x \leq y \Rightarrow f(x) \leq f(y), \forall x \in \mathbb{R}$ (aka monotonically increasing)
- Is order preserving
  - Let $f(x) = x$; arguments and results are identical so order is preserved. Trivial
- Is not order preserving
  - $f(x) = 1/x$
metric-preserving $\iff |x-y| = |f(x)-f(y)|$
- Is order preserving
  - $f(x) = x$ Trivial
- Is not order preserving
  - $f(x) = 1/x$
addition-preserving $\iff f(x+y) = f(x) + f(y)$
- Is addition preserving
  - $f(x) = x$ Trivial
- Is not addition preserving
  - $f(x) = 1/x$

Quaere: Is the identity $f$ always preserving?

Fong, B., & Spivak, D. (2019). An Invitation to Applied Category Theory: Seven Sketches in Compositionality. Cambridge: Cambridge University Press.↩︎

A minor lubridate snip

Sat, 01 Aug 2020 00:00:00 +0000

Dates can be fussy. (Especially when they come to you mangled by manual formatting in spreadsheets.)

Sometimes it’s easier, when there are consecutive dates, simply to create them with seq. For ts, don’t even bother, just use object <- ts(source, start = c(2018,1 ) ...)

The date can be unpacked multiple ways to make presentation easier. Here’s what can be done with days of the week.

suppressPackageStartupMessages(library(lubridate))
seq(c(ISOdate(2020,7,19)), by = "day", length.out = 7) -> the_days

wday(the_days)

## [1] 1 2 3 4 5 6 7

wday(the_days, label = TRUE)

## [1] Sun Mon Tue Wed Thu Fri Sat
## Levels: Sun < Mon < Tue < Wed < Thu < Fri < Sat

wday(the_days, label = TRUE, abbr = FALSE)

## [1] Sunday    Monday    Tuesday   Wednesday Thursday  Friday    Saturday 
## 7 Levels: Sunday < Monday < Tuesday < Wednesday < Thursday < ... < Saturday

# locale argument is OS specific
wday(the_days, label = TRUE, abbr = FALSE, locale = "fr_FR.utf8")

## [1] dimanche lundi    mardi    mercredi jeudi    vendredi samedi  
## 7 Levels: dimanche < lundi < mardi < mercredi < jeudi < ... < samedi

Look before you count

Fri, 31 Jul 2020 00:00:00 +0000

Often, it is possible to back into a number from other numbers, basic K12 learning; we got it pretty well, not knowing any better.

Like all learning, though, it is perishable, as I learned preparing a recent short take.

This all began with a question about data wrangling and how to combine Census data with Bureau of Labor Statistics data to estimate how many residents of Maryland had lost health insurance coverage as a result of COVID-19 related unemployment.

Seems obvious – a person was the unit of analysis, no?

Wrong call. The consequent water boarding of the data would yield no confession. It was not until reframing the question as

How many employment related health insurance policies lapsed?

was progress made.

It was not possible to go from a count of persons employed in 2018 who had employment related insurance coverage directly to the answer. For one thing, some beneficiaries of coverage were dependents, and their own employment status did not change.

It was not possible to use the Bureau of Labor Statistics Community Survey of Employment because it was based on reports by business establishments of changes in positions filled. Due to part-time positions and interstate commutes employment estimates for state residents could not be directly derived.

This brought the job inquiry down to how many fewer Maryland residents had jobs in May 2020 compared to 2018? Data was available on the same basis for both. The difference was approximately 290K fewer persons employed. Each of those can be assumed to have lost the related coverage. And each other person covered under the policy also lost coverage. Those were the lost policies.

Further assumptions were needed to arrive at the estimate of approximately 337K residents who lost coverage, which is a reasonable upper limit.

Why the delay?

In approaching any analysis from how do I do X, rather than what do I want to know, the risk is that the answer to how masks a mis-specification of what.

The lesson re-learned is validate the question first.

Even more wrong

Thu, 02 Jan 2020 00:00:00 +0000

Woody Hayes was supposed to have run up the score 42-12 late in the third quarter and went for a two-point conversion, shortly before the clock ran out. Asked why, he was said to have answered

There are no 3-point conversions!

In the same spirit, I posted the other day about a tweet

“There is no bubble”

via DB pic.twitter.com/MnMSpE38q2
— Daniel Lacalle (@dlacalle_IA) December 30, 2019

propagating a plot was attributed to Deutche Bank Research purporting to show a linear regression of the weekly change in Federal Reserve Bank assets against the weekly change in the S&P Index.

I ran the model with data from Fred for the paltry number of weeks in 2019Q4.

For snicks, I’m now taking a look at $n = 52$, the entire year.

suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(ggthemes)) 
suppressPackageStartupMessages(library(readr)) 

fed <- read_csv("https://gist.githubusercontent.com/technocrat/742747b30ea4c0270f5dbdda0db95a36/raw/47b49a188273af3aed2068017d30b7522500fd23/walcl_2019_PCH.csv")

## Parsed with column specification:
## cols(
##   DATE = col_date(format = ""),
##   WALCL_PCH = col_double()
## )

sp <- read_csv("https://gist.githubusercontent.com/technocrat/d3dc120528026a634c7a9cbd223ff003/raw/c5127108739298743ad6ee16ce937c49ab321b22/sp500_2019_PHC.csv")

## Parsed with column specification:
## cols(
##   DATE = col_date(format = ""),
##   SP500_PCH = col_double()
## )

fed_v_sp <- inner_join(sp, fed, by = "DATE")
fit <- lm(formula = WALCL_PCH ~ SP500_PCH, data = fed_v_sp)
slope <- fit$coefficients[2]
r2 <- summary(fit)$r.squared
rsq <- paste("R squared = ",round(r2,5))
sources <- "Sources:
WALCL: https://fred.stlouisfed.org/series/WALCL
SP500: https://fred.stlouisfed.org/series/SP500
https//technocrat.rbind.io/2019-12-30-not-even-wrong/" 
heading <- "Fed Balance Sheet vs S&P 500\nJan 2 2018-Dec 18, 2019"
y_lab <- "Weekly percentage change in S&P 500 index"
x_lab <- "Weekly percentage change in Federal Reserve Bank assets"
p <- ggplot(fed_v_sp, aes(SP500_PCH, WALCL_PCH))
my_model_plot <- p + geom_point() + geom_abline(slope = slope) + annotate("text", x = 0.15, y = -0.6, label = sources, hjust = 0) + annotate("text", x = -0.25, y = -0.75, label = rsq) + labs(x = x_lab, y = y_lab) + theme_economist() + ggtitle(heading)

If Deutche Bank Research’s mission statement

Deutsche Bank Research is responsible for economic analysis within Deutsche Bank Group and covers asset allocation and all major industry sectors. We analyse relevant trends for the bank in financial markets, the economy and society, highlight risks and opportunities and act as consultant for the bank, its clients and stakeholders.

Deutsche Bank Research delivers high-quality, independent analysis and actively promotes public debate on economic, fiscal, labour market and social policy issues.

is a an accurate reflection of the plot in the original tweet, then obviously its attribution was in error. On the other hand … . Alternatives are left as an exercise for the reader.

Not even wrong

Mon, 30 Dec 2019 00:00:00 +0000

A recent tweet

“There is no bubble”

via DB pic.twitter.com/MnMSpE38q2
— Daniel Lacalle (@dlacalle_IA) December 30, 2019

was commented on by Nassim Nicholas Taleb.

The plot was attributed to Deutche Bank Research.

Our ever-helpful friend Fred has two data series, SP500_PCH and WALCL_PCH that cover the same ground, but only up to 2019-12-18, the latest date for which the S&P data was available as of 2019-12-30.

So, let’s see if we can replicate the DB results:

suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(ggthemes)) 
suppressPackageStartupMessages(library(readr)) 

fed <- read_csv("https://gist.githubusercontent.com/technocrat/d2b231087f35ce987981c98ed5c90042/raw/f266d593bd33c72bcc8f1cb8c0c6415dd6e15fbb/WALCL.csv")

## Parsed with column specification:
## cols(
##   DATE = col_date(format = ""),
##   WALCL_PCH = col_double()
## )

sp <- read_csv("https://gist.githubusercontent.com/technocrat/b93617e7574ff88ddeeb815af529b38c/raw/b9453495b2cf7a55af1ee79806477c0c896eb9f7/SP500_PCH.csv")

## Parsed with column specification:
## cols(
##   DATE = col_date(format = ""),
##   SP500_PCH = col_double()
## )

fed_v_sp <- inner_join(sp, fed, by = "DATE")
fit <- lm(formula = WALCL_PCH ~ SP500_PCH, data = fed_v_sp)
slope <- fit$coefficients[2]
r2 <- summary(fit)$r.squared
rsq <- paste("R squared = ",round(r2,5))
sources <- "Sources:
WALCL: https://fred.stlouisfed.org/series/WALCL
SP500: https://fred.stlouisfed.org/series/SP500
https//technocrat.rbind.io/2019-12-30-not-even-wrong/" 
heading <- "Fed Balance Sheet vs S&P 500\nOct 9-Dec 18, 2019"
y_lab <- "Weekly percentage change in S&P 500 index"
x_lab <- "Weekly percentage change in Federal Reserve Bank assets"
p <- ggplot(fed_v_sp, aes(SP500_PCH, WALCL_PCH))
my_model_plot <- p + geom_point() + geom_abline(slope = slope) + annotate("text", x = 0.15, y = -0.6, label = sources, hjust = 0) + annotate("text", x = -0.25, y = -0.75, label = rsq) + labs(x = x_lab, y = y_lab) + theme_economist() + ggtitle(heading)

What we used to call a wire house (albeit one whose affiliate is known to have made poorly secured real estate loans) is right and your blogger is wrong, or v.v. or both. One of them may also be bullshit, a proposition offered, according to the philospher Harry Frankfurt not for its truth but simply to influence. See On Bullshit.

The only way to tell, is by the maxim of St. Ronald доверяй, но проверяй by one’s own replication. After all, it’s hard to exclude the possibility that neither model is well-formed, and, more importantly, that neither may be particularly useful as an insight.

It’s also worth taking a look at the p-value of the F-statistic for my model: 0.281 – not even close to the $\alpha = 0.05$ laugh test.

What happened to the streak? rle: run length encoding

Thu, 26 Dec 2019 00:00:00 +0000

A common task is to isolate records when some condition changes. Failure analysis is an obvious example. When working in a tidy environment, this isn’t always easy because of its strong column bias. Looking back at previous rows is clunky, at best.

The sometimes overlooked which() provides one part of the solution. Consider a simple case of two vectors, easily combinable into a data frame, one of which indicates a classification and the other, whether or not that classification is present.

genres <- c("Action", "Animation", "Comedy", "Documentary", "Romance", "Short")
row_vector <- c(0,0,1,1,0,0,0)
indices <-min(which(row_vector == TRUE))
indices

## [1] 3

Both genres and row_vector are equal-length character and numeric vectors and the correspondence depends on position, with the Action 0 pair being the first and Romance 0 the last. Python users will recognize this as a hash.

The indices atomic (single element) vector applies, working from the inside out, the which() function to select the elements of row_vector that are TRUE, which evaluates to 1 and min finds the position of the first among that subset. So, we end up with 3 and genres[3] evaluates to Comedy.

Using the positions of elements in one vector to identify elements in another provides a way to use rle(), the run-length encoding function.

What rle does is to keep track of the number of times an element in a vector appears repeated zero or more times.

As usual, it helps to run the help() example, with some inspection:

x <- rev(rep(6:10, 1:5))
x

##  [1] 10 10 10 10 10  9  9  9  9  8  8  8  7  7  6

y <- rle(x)
y

## Run Length Encoding
##   lengths: int [1:5] 5 4 3 2 1
##   values : int [1:5] 10 9 8 7 6

str(y)

## List of 2
##  $ lengths: int [1:5] 5 4 3 2 1
##  $ values : int [1:5] 10 9 8 7 6
##  - attr(*, "class")= chr "rle"

The two pieces of y, y$lengths (or y[1]) and y$values tell us that there are five repetitions of 10, four of 9, etc.

Let’s create a simple data frame

suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tibble)) 
my_data <- tibble(type=rep(c(1:2), each= 9), hour= rep(1:9, 2), event = c(1,1,1,0,0,0,0,1,1,1,1,0,0,0,1,0,1,0))
my_data

## # A tibble: 18 x 3
##     type  hour event
##    <int> <int> <dbl>
##  1     1     1     1
##  2     1     2     1
##  3     1     3     1
##  4     1     4     0
##  5     1     5     0
##  6     1     6     0
##  7     1     7     0
##  8     1     8     1
##  9     1     9     1
## 10     2     1     1
## 11     2     2     1
## 12     2     3     0
## 13     2     4     0
## 14     2     5     0
## 15     2     6     1
## 16     2     7     0
## 17     2     8     1
## 18     2     9     0

Let 1 in column event indicate success and 0 failure. Where does each string of successes turn into failure?

runs <- rle(my_data$event)
runs <- tibble(runs$lengths, runs$values)
colnames(runs) <- c("lengths", "values")
runs

## # A tibble: 8 x 2
##   lengths values
##     <int>  <dbl>
## 1       3      1
## 2       4      0
## 3       4      1
## 4       3      0
## 5       1      1
## 6       1      0
## 7       1      1
## 8       1      0

sequences <- sequences <- tibble(lengths = runs$lengths, values = runs$values) %>% mutate(indices = cumsum(runs$lengths))
sequences

## # A tibble: 8 x 3
##   lengths values indices
##     <int>  <dbl>   <int>
## 1       3      1       3
## 2       4      0       7
## 3       4      1      11
## 4       3      0      14
## 5       1      1      15
## 6       1      0      16
## 7       1      1      17
## 8       1      0      18

post_zero <- sequences %>%  filter(values == 0)
post_zero

## # A tibble: 4 x 3
##   lengths values indices
##     <int>  <dbl>   <int>
## 1       4      0       7
## 2       3      0      14
## 3       1      0      16
## 4       1      0      18

result <- left_join(sequences, post_zero, by = "indices") %>% select(1:3) %>% filter(values.x == 1)
colnames(result) <- c("lengths", "runs", "indices")
result

## # A tibble: 4 x 3
##   lengths  runs indices
##     <int> <dbl>   <int>
## 1       3     1       3
## 2       4     1      11
## 3       1     1      15
## 4       1     1      17

my_data[result$indices,]

## # A tibble: 4 x 3
##    type  hour event
##   <int> <int> <dbl>
## 1     1     3     1
## 2     2     2     1
## 3     2     6     1
## 4     2     8     1

The variable type = 1 had one string of successes that ended at hour three, type = 2 had three ending at hours two, six and eight.

More interesting, of course, is the case where hour is a datetime object and you can bring date arithmetic into play.

The main point is that if you can design a logical test to mutate a numeric column, rle provides a straightforward way of subsetting sequences based on the the test result.

HSL in R, Chapter 1

Sat, 21 Dec 2019 00:00:00 +0000

An R Companion to Chapter 1 of Hosmer, Lemeshow and Sturdivant’s Applied Linear Regression

This post is an R illustration of the first chapter of Hosmer, David W., Stanley Lemeshow, and Rodney X. Sturdivant. Applied logistic regression. Hoboken, New Jersey: Wiley, 2013, the standard text (HLS). Chapter 1 introduces the logistic regression model, how to fit one, test the significance of its coefficients and calculate its confidence intervals. It also discusses other estimation methods, and introduces the datasets used in the book’s examples and exercises. There are three exercises. The chapter is 33 pages long.

Source data

The datasets are available as zip files at the publisher’s data website with 9780470582473 as the search term. The unzipped .txt files are tab delimited. It is assumed that readers are familiar with importing such datasets into R workspaces using tools such as readr::read_tsv(). Note that it will be convenient to bulk download and unzip the files into subdirectories of your workspace, due to the organization of the data website.

The example code assumes that a directory named data is a direct subdirectory of your working directory and that each dataset is in a sudirectory of data, as follows:

data
├── APS
    │   ├── APS_Code_Sheet.pdf
    │   └── APS.txt
    ├── BURN
    │   ├── BURN1000.txt
    │   ├── BURN13M.txt
    │   ├── BURN_Code_Sheet.pdf
    │   ├── BURN_EVAL_1.txt
    │   └── BURN_EVAL_2.txt
    ├── CHDAGE
    │   ├── CHDAGE_Code_Sheet.pdf
    │   └── CHDAGE.txt
    ├── GLOW
    │   ├── GLOW11M.txt
    │   ├── GLOW500.txt
    │   ├── GLOW_BONEMED.txt
    │   ├── GLOW_Code_Sheet.pdf
    │   ├── GLOW_MIS_Comp_Data.txt
    │   ├── GLOW_MIS_WMissing_Data.txt
    │   └── GLOW_RAND.txt
    ├── ICU
    │   ├── ICU_Data_Code_Sheet.pdf
    │   └── ICU.txt
    ├── LOWBWT
    │   ├── LOWBWT_Code_Sheet.pdf
    │   └── LOWBWT.txt
    ├── MYOPIA
    │   ├── MYOPIA_Code_Sheet.pdf
    │   └── MYOPIA.txt
    ├── NHANES
    │   ├── NHANES_Code_Sheet.pdf
    │   └── NHANES.txt
    ├── POLYPHARM
    │   ├── POLYPHARM_Code_Sheet.pdf
    │   └── POLYPHARM.txt
    ├── Scale Example
    │   ├── Scale.Example.code.sheet.docx
    │   ├── Scale.Example.code.sheet.pdf
    │   ├── Scale_Example.out
    │   ├── Scale_Example.tab.txt
    │   ├── Scale_Example.txt
    │   └── Scale_Example.xlsx

Code for text tables and figures

Section 1.1

Table 1.1

Table 1.1 displays the 100 records of the CHDAGE dataset with a constructed age group column, AGEGRP.

chdage <- readr::read_tsv("data/CHDAGE/CHDAGE.txt")

# Classify ages into 5- or 10-year cohorts

df1.1 <-  chdage %>% mutate(cohort = ifelse(AGE < 30,1, 0))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 30 & AGE <= 34,2, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 30 & AGE <= 34,2, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 35 & AGE <= 39,3, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 40 & AGE <= 44,4, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 45 & AGE <= 49,5, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 50 & AGE <= 54,6, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 55 & AGE <= 59,7, cohort))
df1.1 <- df1.1 %>% mutate(cohort = ifelse(AGE >= 60 & AGE <= 90,8, cohort)) 

# use medians of cohort endpoints for plotting
    # medians obtained by hand using md function
    # 
    df1.2 <- df1.1 %>% mutate(plotpoint = ifelse(cohort == 1,24.5, 0))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 2,32, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 3,37, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 4,42, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 5,47, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 6,52, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 7,57, plotpoint))
    df1.2 <- df1.2 %>% mutate(plotpoint = ifelse(cohort == 8,65, plotpoint))

    # Create a text column for cohort lables

    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 1,"20-29", 0))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 2,"30-34", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 3,"35-39", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 4,"40-44", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 5,"45-49", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 6,"50-54", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 7,"55-59", AGEGRP))
    df1.2 <- df1.2 %>% mutate(AGEGRP = ifelse(cohort == 8,"60-70", AGEGRP))

    # split CHD column

    df1.2 <- df1.2 %>% mutate(absent  = ifelse(CHD == 0,1,0))
    df1.2 <- df1.2 %>% mutate(present = ifelse(CHD == 1,1,0))

ID	AGE	AGEGRP	CHD
1	20	20-29	0
2	23	20-29	0
3	24	20-29	0
5	25	20-29	1
4	25	20-29	0
7	26	20-29	0
6	26	20-29	0
9	28	20-29	0
8	28	20-29	0
10	29	20-29	0
11	30	30-34	0
13	30	30-34	0
16	30	30-34	1
14	30	30-34	0
15	30	30-34	0
12	30	30-34	0
18	32	30-34	0
17	32	30-34	0
19	33	30-34	0
20	33	30-34	0
24	34	30-34	0
22	34	30-34	0
23	34	30-34	1
21	34	30-34	0
25	34	30-34	0
27	35	35-39	0
26	35	35-39	0
29	36	35-39	1
28	36	35-39	0
30	36	35-39	0
32	37	35-39	1
33	37	35-39	0
31	37	35-39	0
35	38	35-39	0
34	38	35-39	0
37	39	35-39	1
36	39	35-39	0
39	40	40-44	1
38	40	40-44	0
41	41	40-44	0
40	41	40-44	0
43	42	40-44	0
44	42	40-44	0
45	42	40-44	1
42	42	40-44	0
46	43	40-44	0
47	43	40-44	0
48	43	40-44	1
50	44	40-44	0
51	44	40-44	1
52	44	40-44	1
49	44	40-44	0
53	45	45-49	0
54	45	45-49	1
55	46	45-49	0
56	46	45-49	1
59	47	45-49	1
58	47	45-49	0
57	47	45-49	0
60	48	45-49	0
62	48	45-49	1
61	48	45-49	1
65	49	45-49	1
63	49	45-49	0
64	49	45-49	0
67	50	50-54	1
66	50	50-54	0
68	51	50-54	0
70	52	50-54	1
69	52	50-54	0
71	53	50-54	1
72	53	50-54	1
73	54	50-54	1
75	55	55-59	1
76	55	55-59	1
74	55	55-59	0
78	56	55-59	1
77	56	55-59	1
79	56	55-59	1
82	57	55-59	1
84	57	55-59	1
80	57	55-59	0
85	57	55-59	1
81	57	55-59	0
83	57	55-59	1
86	58	55-59	0
87	58	55-59	1
88	58	55-59	1
90	59	55-59	1
89	59	55-59	1
91	60	60-70	0
92	60	60-70	1
93	61	60-70	1
94	62	60-70	1
95	62	60-70	1
96	63	60-70	1
98	64	60-70	1
97	64	60-70	0
99	65	60-70	1
100	69	60-70	1

Figure 1.1

Figure 1.1 is a scatterplot of coronary heart disease against age in the CHDAGE dataset.

produced with

p1.1 <- ggplot(data = chdage, aes(x = AGE, y = CHD)) + geom_point() + theme_tufte()

Commentary: It’s also instructive to look at the results of

fit1.1 <- lm(CHD ~ AGE, data = chdage)

in a plot produced with

par(mfrow=c(2,2))
plot(fit1.1)

compared to a plot from the mtcars dataset of mpg ~ hp

Table 1.2

Table 1.2 shows the frequency and means of CHD (coronary heart disease)in the CHDAGE dataset by age cohort, produced as follows:

    # Create table with plotting information
    # 
    means_tab <- df1.2 %>% group_by(AGEGRP) %>%  
      summarize(n = n(), Absent = sum(absent),
      Present = sum(present), Mean = mean(CHD),
      plotpoint = mean(plotpoint)) %>% ungroup()

Table 1.3

Table 1.3 presents the result of a logistic model of the CHD against age in the CHDAGE dataset. The equivalent is produced in R by the glm function in {stats} as follows:

chdfit <- glm(CHD ~ AGE, data = chdage, family = binomial(link =
"logit"))
summary(chdfit)

## 
## Call:
## glm(formula = CHD ~ AGE, family = binomial(link = "logit"), data = chdage)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.9718  -0.8456  -0.4576   0.8253   2.2859  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -5.30945    1.13365  -4.683 2.82e-06 ***
## AGE          0.11092    0.02406   4.610 4.02e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 136.66  on 99  degrees of freedom
## Residual deviance: 107.35  on 98  degrees of freedom
## AIC: 111.35
## 
## Number of Fisher Scoring iterations: 4

Note that the standard summary for glm does not produce log likelihood, and it is necessary to call logLik separately.

logLik(chdfit)

## 'log Lik.' -53.67655 (df=2)

The text identifies three methods of calculating confidence intervals for the model fit of the coefficients: the Wald-based confidence interval estimator and the likelihood ratio test on pages 18-19, and the “Score test” on page 14.¹ I refer to these as waldscore, vmscore² and raoscore.

Given a fitted model, two are easy to calculate:

library(MASS)
waldscore <- confint.default(chdfit)
waldscore

##                   2.5 %     97.5 %
## (Intercept) -7.53137370 -3.0875331
## AGE          0.06376477  0.1580775

vmscore <- confint(chdfit)

## Waiting for profiling to be done...

vmscore

##                   2.5 %     97.5 %
## (Intercept) -7.72587162 -3.2461547
## AGE          0.06693158  0.1620067

The third depends on the profileModel package

library(profileModel)
raoscore <- profConfint(profileModel(chdfit, quantile=qchisq(0.95, 1), objective = "RaoScoreStatistic", X = model.matrix(chdfit)))

## Preliminary iteration .. Done
## 
## Profiling for parameter (Intercept) ... Done
## Profiling for parameter AGE ... Done

attr(raoscore, "dimnames")[2][[1]] <- c("2.5 %","97.5 %")
raoscore <- raoscore[,1:2]
raoscore

##                   2.5 %     97.5 %
## (Intercept) -7.50801115 -3.1218722
## AGE          0.06448682  0.1575531

Table 1.4

Table 1.4., the covariance matrix of the estimated coefficients in Table 1.3, is produced by in equivalent form by

vcov(chdfit)

##             (Intercept)           AGE
## (Intercept)  1.28517059 -0.0266769747
## AGE         -0.02667697  0.0005788748

Figure 1.3

Figure 1.3 plots the profile log likelihood for the AGE variable in the CDHAGE dataset. See the discussion on page 19 of HLS. It can be produced with the following code

library(dplyr)
library(purrr)
library(ggplot2)


get_profile_glm <- function(aglm){
  prof <- MASS:::profile.glm(aglm, del = .05, maxsteps = 52)
  disp <- attr(prof,"summary")$dispersion
  purrr::imap_dfr(prof, .f = ~data.frame(par = .y,
                  deviance=.x$z^2*disp+aglm$deviance, 
                  values = as.data.frame(.x$par.vals)[[.y]], stringsAsFactors = FALSE))
}

chdage <- readr::read_tsv("data/CHDAGE/CHDAGE.txt")

## Parsed with column specification:
## cols(
##   ID = col_double(),
##   AGE = col_double(),
##   CHD = col_double()
## )

chdfit <- glm(CHD ~ AGE, data = chdage, family = binomial(link = "logit"))


pll <- get_profile_glm(chdfit) %>% filter(par == "AGE") %>% mutate(beta = values) %>% mutate(pll = deviance * -0.5) %>% dplyr::select(-c(par,values, deviance))

asymmetry <- function(x) {
  ci <- confint(x, level = 0.95)
  ci_lower <- ci[2,1]
  ci_upper <- ci[2,2]
  coeff <- x$coefficients[2]
  round(100 * ((ci_upper - coeff) - (coeff - ci_lower))/(ci_upper - ci_lower), 2)
}

asym <- asymmetry(chdfit)

## Waiting for profiling to be done...

ggplot(data = pll, aes(x = beta, y = pll)) + 
  geom_line() + 
  scale_x_continuous(breaks = c(0.06, 0.08, 0.10, 0.12, 0.14, 0.16)) + 
  scale_y_continuous(breaks = c(-57, -56, -55, -54)) + 
  xlab("Coefficient for age") + 
  ylab("Profile log-likelihood function") + 
  geom_vline(xintercept = confint(chdfit)[2,1]) + 
  geom_vline(xintercept = confint(chdfit)[2,2]) + 
  geom_hline(yintercept = (logLik(chdfit) - (qchisq(0.95, df = 1)/2))) + 
  theme_classic() + 
  ggtitle(paste("Asymmetry =", scales::percent(asym/100, accuracy = 0.1))) + theme(plot.title = element_text(hjust = 0.5))

## Waiting for profiling to be done...

## Waiting for profiling to be done...

For a brief discussion of development of the get_profile_glm function, see this stackoverflow post. Note that the arguments to

prof <- MASS:::profile.glm(chdfit, del = .05, maxsteps = 52)

were hand selected specifically for this example and will likely require adjustment for different data.

Commentary

Chapter 1 is foundational. Understanding it adequately is key to further progress. Parsing the equations is part of the task. It is helpful to write them out in an equation formatter, such as $\LaTeX$, which is built into RStudio.

Equation 1.1

To produce

\[\pi(x) = \frac{e^{\beta{_0}+\beta{_1}x}}{1 + e^{\beta{_0}+\beta{_1}x}}\]

and

\[g(x) = ln\Bigl[\frac{\pi(x)}{1 - \pi(x)}\Big] = \beta_0 +\beta_1x\]

write

$$pi(x) = \frac{e^{\beta{_0}+\beta{_1}x}}{1 + e^{\beta{_0}+\beta{_1}$$

and

$$g(x) = ln\Bigl[\frac{\pi(x)6}{1 - \pi(x)}\Big] = \beta_0 +\beta_1x$$

Equation 1.2

\[\pi(x_i)^{y_i}[1 - \pi(x_i)]^{1-y_i}\]

Equation 1.3

\[l\beta = \prod_{i = 1}^n\pi(x)^{y_i}[1 - \pi(x)]^{1-y_i}, n = 200\]

Equation 1.4

\[L(\beta) = ln[l(\beta)] = \sum_{i = 1}^{n}[y_i, ln[\pi(x_i)] + (1 - y_i)ln[1 -\pi(x-i)])\]

Why Score is capitalized is somewhat of a mystery, since it was developed by Rao.↩︎
After Venzon and Moolgavkar, whose underlying methodology is the basis.↩︎

Logistic regression

Wed, 30 Oct 2019 00:00:00 +0000

Why learn about logistic regression?

Because the outcome, $Y$, is binary, rather than continous, we choose logistic regression, rather than linear regression.

Hosmer, David W., Stanley Lemeshow, and Rodney X. Sturdivant. Applied logistic regression. Hoboken, New Jersey: Wiley, 2013, the standard text (HLS).

HSL

HSL doesn’t have code examples, but it does have tables, figures and exercises

My plan is to work those in R and to post the resu1ts here, with the scripts available on github.

Also in preparation

Along side those planned posts is a mini-primer to walk through a real-world dataset.

Metadata for datasets

Fri, 18 Oct 2019 00:00:00 +0000

Where does this dataset come from?

Is this the original revised data or the revised revised data?

Keeping track of the provenance of data can be a challenge, especially when drawing on published sources. Keeping a record of the origin, the date accessed, the transformations applied (e.g., converting from .xls to cvs and converting character strings such as “$1,250,321.21” to floats or date strings to date objects), subsequent changes, who handled the data object and where it can be found in a repository are all things that enhance the analyst’s own ability to reproduce results.

Unfortunately, notes go missing, files get mis-filed and all the other hazards that can befall research can happen. Often, one wishes for R objects with built-in metadata for that purpose.

Using mostattributes() to do attach metadata

Scott Chamerlain at ropensci.org brought attr to my attention, which is the built-in way I was looking for originally. He also pointed me to EML, a much more elaborate approach suited for publication projects.

A minimal example

Create data frame and a separate metadata list

## Loading required package: jsonlite

fips <- read.csv("https://tuva.s3-us-west-2.amazonaws.com/state_fips_postal.csv", header = FALSE)
colnames(fips) = c("state", "fip", 'id')
require(jsonlite) # easier to use JSON to write metadata
meta <- fromJSON("https://tuva.s3-us-west-2.amazonaws.com/2015-07-31-meta.json")

The json source file looks like this

[
    {
    "Accessed": "2015-07-31",
    "GitBlame": "Richard Careaga",
    "Contact": "technocrat@twitter",
    "Preprocessing": "FIPS Codes for the States and District of Columbia table captured manually and converted to cvs file",
    "Source": "https://www.census.gov/geo/reference/ansi_statetables.html",
    "Repository": "unassigned",
    "Version": "1.0"
    }
]

Associate the metadata with the data frame using mostattributes

x <- fips
mostattributes(x) <- list(meta = meta)

Now metadata is displayed by default

## [[1]]
##  [1] "Alabama"              "Alaska"               "Arizona"             
##  [4] "Arkansas"             "California"           "Colorado"            
##  [7] "Connecticut"          "Delaware"             "District of Columbia"
## [10] "Florida"              "Georgia"              "Hawaii"              
## [13] "Idaho"                "Illinois"             "Indiana"             
## [16] "Iowa"                 "Kansas"               "Kentucky"            
## [19] "Louisiana"            "Maine"                "Maryland"            
## [22] "Massachusetts"        "Michigan"             "Minnesota"           
## [25] "Mississippi"          "Missouri"             "Montana"             
## [28] "Nebraska"             "Nevada"               "New Hampshire"       
## [31] "New Jersey"           "New Mexico"           "New York"            
## [34] "North Carolina"       "North Dakota"         "Ohio"                
## [37] "Oklahoma"             "Oregon"               "Pennsylvania"        
## [40] "Rhode Island"         "South Carolina"       "South Dakota"        
## [43] "Tennessee"            "Texas"                "Utah"                
## [46] "Vermont"              "Virginia"             "Washington"          
## [49] "West Virginia"        "Wisconsin"            "Wyoming"             
## 
## [[2]]
##  [1]  1  2  4  5  6  8  9 10 11 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28
## [26] 29 30 31 32 33 34 35 36 37 38 39 40 41 42 44 45 46 47 48 49 50 51 53 54 55
## [51] 56
## 
## [[3]]
##  [1] "AL" "AK" "AZ" "AR" "CA" "CO" "CT" "DE" "DC" "FL" "GA" "HI" "ID" "IL" "IN"
## [16] "IA" "KS" "KY" "LA" "ME" "MD" "MA" "MI" "MN" "MS" "MO" "MT" "NE" "NV" "NH"
## [31] "NJ" "NM" "NY" "NC" "ND" "OH" "OK" "OR" "PA" "RI" "SC" "SD" "TN" "TX" "UT"
## [46] "VT" "VA" "WA" "WV" "WI" "WY"
## 
## attr(,"meta")
##     Accessed        GitBlame            Contact
## 1 2015-07-31 Richard Careaga technocrat@twitter
##                                                                                          Preprocessing
## 1 FIPS Codes for the States and District of Columbia table captured manually and converted to cvs file
##                                                       Source Repository Version
## 1 https://www.census.gov/geo/reference/ansi_statetables.html unassigned     1.0

Missing Inaction--locating data holes

Wed, 18 Sep 2019 00:00:00 +0000

## Note: Using an external vector in selections is ambiguous.
## ℹ Use `all_of(interest_vars)` instead of `interest_vars` to silence this message.
## ℹ See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
## This message is displayed once per session.

Your hands can’t hit what your eyes can’t see. Muhammad Ali

In real-world data, there is often so much to be re-arranged, re-formatted and supplemented that it’s easy to overlook what isn’t there. I wrote about this recently, but decided it could be improved.

Some real world data, masked and anonymous

To illustrate some of the ways that a dataset can snooker you, I’ll be using a tibble with 22,471 rows and 38 columns. That’s 853,898 slots for values, far beyond eyeballing’s reach.

This observational data is to be used for a logistic model of VAR30 and VAR31 as the alternative dependent variables. The continuous variables are VAR1 (simply a record id), VAR2 and VAR7. All others are dichotomous, coded 1 for present and 0 for absent. VAR14-VAR29 are the principal independent variables of interest.

Here is the data layout.

## tibble [22,471 × 38] (S3: tbl_df/tbl/data.frame)
##  $ VAR1 : int [1:22471] 420893 303936 191488 602555 319182 975171 327170 820841 385483 996417 ...
##  $ VAR2 : num [1:22471] 36 22 17 39 64 54 46 45 44 16 ...
##  $ VAR3 : num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR4 : num [1:22471] 0 0 1 0 0 0 0 0 0 1 ...
##  $ VAR5 : num [1:22471] 1 1 0 1 1 1 1 1 1 0 ...
##  $ VAR6 : num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR7 : num [1:22471] 2008 2008 2008 2008 2008 ...
##  $ VAR8 : num [1:22471] 1 0 0 0 0 0 0 0 0 0 ...
##  $ VAR9 : num [1:22471] 0 1 1 1 1 1 1 1 1 1 ...
##  $ VAR10: num [1:22471] 1 0 1 1 0 1 0 1 1 1 ...
##  $ VAR11: num [1:22471] 1 0 1 1 1 1 1 1 1 1 ...
##  $ VAR12: num [1:22471] 0 1 0 0 0 0 0 0 0 0 ...
##  $ VAR13: num [1:22471] 1 1 1 1 1 1 1 1 1 0 ...
##  $ VAR14: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR15: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR16: num [1:22471] 0 0 0 0 0 0 1 0 0 1 ...
##  $ VAR17: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR18: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR19: num [1:22471] 0 0 0 0 0 0 1 0 0 0 ...
##  $ VAR20: num [1:22471] 0 0 0 0 0 0 0 0 0 1 ...
##  $ VAR21: num [1:22471] 0 0 0 0 1 1 0 0 0 0 ...
##  $ VAR22: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR23: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR24: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR25: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR26: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR27: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR28: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR29: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR30: num [1:22471] 1 0 0 1 1 1 1 0 1 1 ...
##  $ VAR31: num [1:22471] 0 1 1 0 0 0 0 1 0 0 ...
##  $ VAR32: num [1:22471] 1 0 1 1 0 0 0 0 0 0 ...
##  $ VAR33: num [1:22471] 0 1 0 0 1 1 1 1 1 1 ...
##  $ VAR34: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR35: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR36: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR37: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...
##  $ VAR38: num [1:22471] 0 0 0 0 0 0 0 0 0 0 ...

What’s missing?

The not applicables

To see if a data frame or tibble contains any missing values, the handy function is anyNA(). For our data, the result is TRUE.

A quick way to identify how many missing values there are is CountCompCases{DescTools}. (There is also complete.cases{base}, but it returns a logical list, rather than a summary.)

## 
## Total rows:      22471
## Complete Cases:  22052 (98.1%)
## 
##     vname  nas  nas_p  cifnot  cifnot_p
## 1    VAR1    0   0.0%   22052     98.1%
## 2    VAR2    0   0.0%   22052     98.1%
## 3    VAR3    0   0.0%   22052     98.1%
## 4    VAR4    0   0.0%   22052     98.1%
## 5    VAR5    0   0.0%   22052     98.1%
## 6    VAR6    0   0.0%   22052     98.1%
## 7    VAR7    0   0.0%   22052     98.1%
## 8    VAR8    0   0.0%   22052     98.1%
## 9    VAR9    0   0.0%   22052     98.1%
## 10  VAR10    0   0.0%   22052     98.1%
## 11  VAR11    0   0.0%   22052     98.1%
## 12  VAR12    0   0.0%   22052     98.1%
## 13  VAR13    0   0.0%   22052     98.1%
## 14  VAR14    0   0.0%   22052     98.1%
## 15  VAR15    0   0.0%   22052     98.1%
## 16  VAR16    0   0.0%   22052     98.1%
## 17  VAR17    0   0.0%   22052     98.1%
## 18  VAR18    0   0.0%   22052     98.1%
## 19  VAR19    0   0.0%   22052     98.1%
## 20  VAR20    0   0.0%   22052     98.1%
## 21  VAR21    0   0.0%   22052     98.1%
## 22  VAR22    0   0.0%   22052     98.1%
## 23  VAR23    0   0.0%   22052     98.1%
## 24  VAR24    0   0.0%   22052     98.1%
## 25  VAR25    0   0.0%   22052     98.1%
## 26  VAR26    0   0.0%   22052     98.1%
## 27  VAR27    0   0.0%   22052     98.1%
## 28  VAR28    0   0.0%   22052     98.1%
## 29  VAR29    0   0.0%   22052     98.1%
## 30  VAR30    0   0.0%   22052     98.1%
## 31  VAR31    0   0.0%   22052     98.1%
## 32  VAR32  419   1.9%   22052     98.1%
## 33  VAR33  419   1.9%   22052     98.1%
## 34  VAR34  419   1.9%   22052     98.1%
## 35  VAR35  419   1.9%   22052     98.1%
## 36  VAR36  419   1.9%   22052     98.1%
## 37  VAR37  419   1.9%   22052     98.1%
## 38  VAR38    0   0.0%   22052     98.1%

We see that 419 have NA values, about 1.9% of all observations.

Why do we care?

NAs are squirrely when it comes to computation.¹

NA+1 = NA
NA-1 = NA
NA*1 = NA
NA/1 = NA
NA^1 = NA
NA^0 = 1
NA**2 = NA

More likely than not, leaving them will throw errors.

What can we do?

We have two alternatives: remove the rows with NAs or impute values. Because the NAs are only 1.9% of the data, I’m going to avoid all the skull sweat needed for good imputation (substituting some summary statistic for the NA). An additional consideration is that the variables with the NA are a bit dodgy to begin with for reasons not germane to the subject. They also turned out not to contribute to the modeling.

How do we remove the NAs?

The dplyr filter function takes a logical argument and !isNA() provides one.

mydata_no_na <- mydata %>% filter(!is.na(VAR32))

A little sleight of hand was involved. I guessed that all of the variables with NA would be missing in each observation, so that by filtering on one, I’d hit them all. Sure enough, anyNA() confirmed my guess. If it hadn’t, of course, I would have needed to either run multiple filters or expand to

mydata_no_na <- mydata %>% filter(!is.na(VAR32) | !is.na(VAR33) ...)

But wait!

Dependent variables of interest are all zero valued for some observations

These data were assembled for a purpose – to determine the influence of a group of 16 variables (different types of medical trauma) on 2 variables considered separately (different types of internal injuries to the same organ). Observations in which none of the 16 variables were coded as present (1) shed no light on the problem.

How do we identify them?

Recall that each of the 16 variables is either valued as 1 or 0. If the sum of the variables for an observation is zero, we don’t need the observation for the main purposes of the model project. There are several possible ways to sum variables rowwise. I chose a little linear algebra jujitsu.

Linear algebra!!! Relax, I don’t know much about this subject either. But I know a piece that’s simple and well suited to this problem.

Starting with mydata_no_na, the first step is to create an object for the 16 variables

    interest_vars <- c("VAR14", "VAR15", "VAR16", "VAR17", "VAR18", "VAR19", "VAR20", "VAR21", "VAR22", "VAR23", "VAR24", "VAR25", "VAR26", "VAR27", "VAR28", "VAR29")

and to use it to subset the NAs removed data

    mydata_vars <- mydata_no_na %>% dplyr::select(interest_vars)

Here comes the trick: It’s easy to convert a tibble into a matrix.

    m <- as.matrix(mydata_vars)

Why do we want a matrix? A matrix is an object that you can perform row operations on easily.

    rowtots <- rowSums(m)

We now have the sum of the 16 variables for each observation.

The return trip is equally easy.

    vcode_count <- enframe(rowtots) %>% dplyr::select(value) %>% dplyr::rename(V_COUNT = value)
    mydata_w_row_count <- as_tibble(cbind(mydata_no_na, vcode_count))
    mydata_singlets <- mydata_with_row_count %>% filter(V_COUNT == 1)

In short, we’ve added a row total field, V_COUNT and filtered for sums of 1.

The results are dramatic. We started with 22,471 observations, subtracted 419 NA, and then isolated the observations with one (only one) of the 16 variables and ended up with 9,114 observations.

Even more

We’ve dealt with NA, eliminated observations with 0 or more than 1 occurrences. There is another possibility, observations with more than one variable of interest. These are easy to identify

        mydata_multi <- mydata_with_row_count %>% filter(V_COUNT > 1)

and there are 2,954 observations. These are candidates for a separate model because there are potentially multiple interacting independent variables that influence the dependent variables. (Or they could represent inconsistencies in data collection.)

Flash cards

Don’t eyeball
Use anyNA()
Check for “useless” observations
Check for observations needing different treatment

“Numerical computations using NA will normally result in NA: a possible exception is where NaN is also involved, in which case either might result (which may depend on the R platform). Logical computations treat NA as a missing TRUE/FALSE value, and so may return TRUE or FALSE if the expression does not depend on the NA operand.” [NA]{base}↩︎

Hidden missing data

Sat, 24 Aug 2019 00:00:00 +0000

No, you can’t always get what you want
You can’t always get what you want
You can’t always get what you want
But if you try sometime
You’ll find
You get what you need – Mick Jagger and Keith Richards

There’s nothing like missing values to mess up a model. But you have to work with what you get. The data set is not often coded with NAs where data would be for an observation of a variable, sometimes it is blank.

For that, we have complete.cases(){stats} which will subset the data to include only which rows have no missing data.

More insidious is the case where observations for a dichotomous variable, that can take on only one of two possible values. These may be coded as TRUE/FALSE or 1/0. And, of course, if the coding is complete, there are no missing values. That doesn’t mean that you have what you need.

Recently, I worked on a largish data set that ended up as approximately 22K records with 38 variables. The problem arose in a set of 16 independent dichotomous variables – call them $Var1 ... Var16$ – being used to model a binary response variable. This part of the data looks like

hidden
# A tibble: 22,471 x 16
    VAR1  VAR2  VAR3  VAR4  VAR5  VAR6  VAR7  VAR8  VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
 1     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
 2     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
 3     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
 4     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
 5     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0
 6     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0
 7     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0
 8     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
 9     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0
10     0     0     1     0     0     0     1     0     0     0     0     0     0     0     0     0
# … with 22,461 more rows

First problem: The purpose of the logistic regression analysis was to see which of the sixteen variables were associated with the response variable. The first four observations (rows), at least, had all zeros. It is not until the fifth row when we hit a positive observation, in $VAR8$.

But cockroaches are not hermits. In row 10, there are observations for $VAR3$ and $VAR7$. The interactions between two or more variables is an interesting problem, but it was out-of-scope for this stage of analysis if for no other reason than it was not clear whether they might be mutually exclusive.

The net result is we have some observations with t00 few and some with too many variables.

Neo to the rescue. There a R objects, like vectors, that you can apply a function to each element

some_function(vector)

and get back a vector where every number has been squared, incremented by 2 or whatever operation the function performs.

Another one of those objects is a matrix. A matrix has one or more rows and one or more column in its simple two dimension case, all of the same class (no mixing numbers and characters). For our purposes, it’s basically a spreadsheet.

How does this help? We can convert our tibble with the rows containing different numbers of variable observations to a matrix, very simply

m <- as.matrix(hidden)
head(m)
      VAR1 VAR2 VAR3 VAR4 VAR5 VAR6 VAR7 VAR8 VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16
[1,]    0    0    0    0    0    0    0    0    0     0     0     0     0     0     0     0
[2,]    0    0    0    0    0    0    0    0    0     0     0     0     0     0     0     0
[3,]    0    0    0    0    0    0    0    0    0     0     0     0     0     0     0     0
[4,]    0    0    0    0    0    0    0    0    0     0     0     0     0     0     0     0
[5,]    0    0    0    0    0    0    0    1    0     0     0     0     0     0     0     0
[6,]    0    0    0    0    0    0    0    1    0     0     0     0     0     0     0     0

Next, we apply the sumCols() function to add up the entries for the rows and assign it.

row_count <- rowSums(m)
head(row_count)
[1] 0 0 0 0 1 1
length(row_count)
[1] 22471

the result is a numeric vector, conveniently of the same length as the number of rows in the tibble, and in the same order.

summed_up <- as_tibble(cbind(hidden, row_count))
summed_up
# A tibble: 22,471 x 17
    VAR1  VAR2  VAR3  VAR4  VAR5  VAR6  VAR7  VAR8  VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16 row_count
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl>
 1     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
 2     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
 3     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
 4     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
 5     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0         1
 6     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0         1
 7     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
 8     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
 9     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0     0         0
10     0     0     1     0     0     0     1     0     0     0     0     0     0     0     0     0         2
# … with 22,461 more row

Now, with a bit of tidyness, we can set a count, using the dplyr package of the rows with varying numbers of entries..

summed_up %>% group_by(row_count) %>% count()
# A tibble: 6 x 2
# Groups:   row_count [6]
  row_count     n
      <dbl> <int>
1         0 10083
2         1  9349
3         2  2509
4         3   461
5         4    63
6         5     6

The bad news is that nearly half of the data are of no help, the 10K rows with none of the sixteen variables having a positive value. The sad news is the 3K rows with multiple variables that we’re not ready to tackle yet.

summed_up %>% filter(row_count > 1)
# A tibble: 3,039 x 17
    VAR1  VAR2  VAR3  VAR4  VAR5  VAR6  VAR7  VAR8  VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16 row_count
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl>
 1     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
 2     0     0     1     0     0     0     1     0     0     0     0     0     0     0     0     0         2
 3     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
 4     0     0     0     0     0     1     0     0     1     0     0     0     0     0     0     0         2
 5     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
 6     0     0     1     0     0     0     0     0     0     0     0     0     0     0     1     0         2
 7     0     1     0     0     0     0     0     0     0     0     1     0     0     0     0     0         2
 8     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
 9     0     0     1     0     0     1     0     0     0     0     0     0     0     0     0     0         2
10     0     0     1     0     0     0     0     1     0     0     0     0     0     0     0     0         2
# … with 3,029 more rows

Now that we know where we stand, we can filter to get the 9K rows with a single variable only. We bind the row_count variable to the original tibble.

row_totals <- summed_up %>% select(row_count)
singlets <- bind_cols(hidden, row_totals) %>% filter(row_count == 1)
singlets
# A tibble: 9,349 x 17
    VAR1  VAR2  VAR3  VAR4  VAR5  VAR6  VAR7  VAR8  VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16 row_count
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl>
 1     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0         1
 2     0     0     0     0     0     0     0     1     0     0     0     0     0     0     0     0         1
 3     0     0     0     0     0     0     0     0     0     0     0     0     0     0     1     0         1
 4     0     0     0     0     0     0     0     0     0     0     0     0     1     0     0     0         1
 5     0     0     0     0     0     1     0     0     0     0     0     0     0     0     0     0         1
 6     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0         1
 7     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0         1
 8     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0         1
 9     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0         1
10     0     0     1     0     0     0     0     0     0     0     0     0     0     0     0     0         1
# … with 9,339 more rows

I should point out a couple of things. The last tibble above is not the complete data set I am working on. Other variables, particularly the response variable are intentionally omitted. I also could have accomplished the same end more economically by piping operations to avoid intermediate object completion. I thought, in this case, it would be easier to follow with the intermediates.

Next time, I’ll describe the logistic model and the different results obtained by applying it to the entire data set, the empty data set, the multiple data set and the single-variable per observation data set.

Oh, some readers who made it this far are yawning – Nothing but basic linear algebra. The readers who have heard of linear algebra and might have been intimidated will see, I hope, that parts of it are easy to grasp, easy to apply, and useful.

Review of R Cookbook, 2nd Ed.

Thu, 18 Jul 2019 00:00:00 +0000

There is a whole sub-culture of food porn, the lavish coffee table books that set out to educate you to an entire culture through its food. I once saw a library of several hundred.

These are page flippers – doesn’t that look marvelous, we’ll have to try it next time we find halibut cheeks. The sad fact, though, is that any of these books are lucky if the owner actually uses one or two of its recipes and exalted if a recipe becomes a dinner party staple.

The first edition of R Cookbook by Paul Teetor, published in 2011, was, in analogy, similar in that it provided flavors with good recipes, many of which proved useful for specific needs.

The second edition, just out with J.D. Long as co-author retraces and updates much of the content of the first edition, brings RStudio and tidy tools to bear, and subverts the dominant paradigm. You flip through it and it first appears to be the cookbook equivalent of granny’s index card recipe box. But it starts off with enough new that you stop flipping pages and start reading.

It took me two days to read the 554 pages. I had the help of several airplane hours, but it would have been a page turner anyway. What captivates is the rigorous R tutorial cleverly hidden in asides, call-out boxes and brief explanations of why a particular code chunk does one thing when you might reasonably expect another.

In my childhood, Donald Duck had three nephews, Huey, Dewey and Louie who got their uncle out of more than one jam by consulting The Junior Woodchuck Manual, which somehow contained the answer to any question that could possibly arise. The R Cookbook, 2ND Ed. comes close to that standard for the beginner to intermediate user who is completely innocent, on the one hand, or who has been covertly using functions without a well-founded understanding of their requirements and limitations.

It would be an unconventional choice, perhaps, but this is the book that I would choose for the text if I were teaching a course in introductory R. Why? Because this book tells you what R does, not what it is. Any student coming to it with even an imprecise notion of $f(x) = y$ is going to be able to follow along to the point, perhaps, of wanting to write her own package. The text doesn’t cover this, but points you in the right direction, and suggests source(“good_stuff.R”) as an interim step.

In other words, its approach is like R’s, functional, not imperative, definitional not procedural. In my view, the implications in the R vs. Python wars are obvious. In one model, Professor Higgens tells Eliza bring me my slippers. In the other, he first has to declare a subject (the slippers), a possessive (my slippers), object (grammatical, me) and then, with the verb, provide detailed instructions on where they are to be found, how brought into possession and the means by which they are to be transported to their destination.

Someone once said something to the effect that assembly language uses humans as pre-processors. I’m afraid that is true of many of its do-this, then do-that descendants.

For now, their defense is the superiority of compiled programs in speed on the metal and the much more expensive wetware. And they are right for the one-off case. But as cases proliferate or become abstracted, wetware costs go up faster than throughput advantages.

It will not happen while I’m still sentient enough to follow, but a compiled R or an R to Haskell API does seem to provide the best of both worlds. Put the wetware where it belongs, in defining the problem and the appropriate solution and leave the optimization to languages that are good at implementing functional logic.

Sed ego et qui me defricatus urina contra ventum?

eDiscovery is pattern recognition and eyeballs are expensive

Mon, 03 Jun 2019 00:00:00 +0000

Clients are transforming their businesses with the tools of data science. How long do you suppose they will pay $50/hour for humans to read tons of ediscovery?

The legal profession owes a great debt to the 19th century mathemetician Georg Boole whose work became the genesis of the boolean algebra that every lawyer is familiar through the use of keyword searches. For decades, it has helped us wade through more than we could read.

Boolean searches are, alas, a form of simple pattern recognition that relatively simple computing techniques can do with greater speed and accuracy at lower cost. Another time, I will write on the more advanced toolsets of natural language processing capable of detecting case relevant patterns that our keyword searches miss.

This time, however, let’s highlight the power of social network analysis. Imagine a room filled with sealled envelopes with nothing more than From and To, universal metadata in emails. By treating each From and each To as nodes, and the envelope as an edge connecting them, it is relatively simple to classify users into groups, and identify the users in those groups on the basis of measures of prominence. Having identified groups in this way, I opened the envelopes and discovered that my three groups had differences in vocabulary. One group’s vocabuary contained approximately 20% of words found in neither of group, one group had 10% and a third group none.

My HarvardX term paper puts the problem in context for my academic audience, and the technical analysis, which was my primary purpose, is decribed in a way that I hope most litigators will find accessible.

As a whole, the legal profession is behind the curve for the application of the applied statistsical methods of data science to its work. It is not difficult to foresee re-arranement of the AMLAW 100 in the coming decade that reflects differences among firms in catching up.

The compiler will tell you what the user cannot

Tue, 08 Jan 2019 00:00:00 +0000

The compiler will always tell you about source code errors that prevent compiling. It can’t advise you if your code solves the problem that it was supposed to solve, even if you are confident in what that problem is. But have a thought for the user who posed the problem, and keep in mind two famous quotations.

Thus, programs must be executed for humans to read, only incidentially for machines to execute. – Harold Abramson, Gerald Jay Sussman, Structure and Interpretation of Computer Programs - 2nd Edition, The MIT Press, 1966

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it? – Brian Kernighan and P.J. Plauger, The Elements of Programming Style - 2nd Edition, McGraw-Hill, 1978

I’ve been helping a medical researcher who was given the following perfectly syntactical R code:


#1. Read in the necessary data.

setwd("~")
ReceivedOp <- list()

for (i in 1:14) { ## csv files - T/F for every group - based on whether or not they received an operation
ReceivedOp[[i]] <- read.csv(paste("./receivedOperationByYear/receivedOperation", (2002:2015)[i], ".csv", sep = ""))
}

rm(i)

filePathsDCODEs <- paste("./RDS AY ", 2002:2015, "/RDS_DCODE.csv", sep = "") ## The names of each file path

D_CODES <- list()

for (i in 1:14) { ## Diagnostic codes (IP or EP)
D_CODES[[i]] <- read.csv(filePathsDCODEs[i])
}

rm(i)

filePathsDEMOs <- paste("./RDS AY ", 2002:2015, "/RDS_DEMO.csv", sep = "") ## The names of each file path

DEMO <- list()

for (i in 1:14) {
DEMO[[i]] <- read.csv(filePathsDEMOs[i])
} ## Demographic data (i.e. ages)

rm(i)

DISCHARGE <- list()

for (i in 1:9) { ## Mortality information by INC_KEY
DISCHARGE[[i]] <- read.csv(paste("./RDS AY ", (2007:2015)[i], "/DISCHARGE", (2007:2015)[i], ".csv", sep = ""))
}

rm(i)

for (i in 6:14) { ## Change integer division to regular division
DEMO[[i]]$AGE[DEMO[[i]]$AGEU == "Days"] <- DEMO[[i]]$AGE[DEMO[[i]]$AGEU == "Days"] / 365
DEMO[[i]]$AGE[DEMO[[i]]$AGEU == "Months"] <- DEMO[[i]]$AGE[DEMO[[i]]$AGEU == "Months"] / 12
DEMO[[i]]$AGEU[DEMO[[i]]$AGEU == "Days" | DEMO[[i]]$AGEU == "Months"] <- "Years"
}

# 2. Collect data of interest

RuptureTypeList <- list()
InteractionTerms <- list()
DemoList <- list()
AgexRuptureTypexOperationTypeInteractions <- list()
AgexOperationTypeInteractions <- list()
AgexRuptureTypexMortalityInteractions <- list()
OrganizedDischarge <- list()
AgexRupturexOpxMort <- list()
pelvicFractures <- list()

for (i in 1:14) { ## Pre-allocate memory
RuptureType <- rep(NA, nrow(ReceivedOp[[i]]))
RuptureTypeList[[i]] <- RuptureType
InteractionTerms[[i]] <- RuptureType
DemoList[[i]] <- RuptureType
AgexRuptureTypexOperationTypeInteractions[[i]] <- RuptureType
AgexOperationTypeInteractions[[i]] <- RuptureType
AgexRuptureTypexMortalityInteractions[[i]] <- RuptureType
OrganizedDischarge[[i]] <- RuptureType
AgexRupturexOpxMort[[i]] <- RuptureType
pelvicFractures[[i]] <- matrix(rep(RuptureType, 16), ncol = 16)
}

rm(RuptureType, i)

for (i in 1:14) { ## The code here is going to determine whether there is an EP or IP rupture, by converting to numeric before checking, difference between 867 and 867.0 is.
for (j in 1:nrow(ReceivedOp[[i]])) { ## Going through by INC_KEY in the ReceivedOp
if (867.1 %in% suppressWarnings(as.numeric(as.character(D_CODES[[i]]$DCODE[D_CODES[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]])))) {
RuptureTypeList[[i]][j] <- 867.1
}
else if (867 %in% suppressWarnings(as.numeric(as.character(D_CODES[[i]]$DCODE[D_CODES[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]])))) {
RuptureTypeList[[i]][j] <- 867
}
}
}

rm(i, j)

for (i in 1:14) { ## Whether they had a bladder operation and whether their rupture was EP or IP.
InteractionTerms[[i]] <- interaction(ReceivedOp[[i]]$BLADDEROP, RuptureTypeList[[i]], drop = T) ## The interaction function is very useful.
}

rm(i)

for (i in 1:14) { ## Transfer each patient's age
for (j in 1:nrow(ReceivedOp[[i]])) {
if (!is.na(DEMO[[i]]$AGE[DEMO[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]][1] > 0) & DEMO[[i]]$AGE[DEMO[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]][1] > (1/12)) {
## Condition as such after examining the data outside the admissable range and finding all values were either 0 or negative (the latter representing missing data)
DemoList[[i]][j] <- DEMO[[i]]$AGE[DEMO[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]][1]
}
}
}

for (i in 1:14) { ## Convert ages into categories
DemoList[[i]] <- ifelse(17 >= DemoList[[i]], "Child", "Adult")
}

for (i in 1:14) { ## Contains information about age, EP vs IP, and whether they received an operation.
AgexRuptureTypexOperationTypeInteractions[[i]] <- interaction(ReceivedOp[[i]]$BLADDEROP, RuptureTypeList[[i]], DemoList[[i]], drop = T)
}

for (i in 1:14) { ## Whether they received an operation and their age
AgexOperationTypeInteractions[[i]] <- interaction(ReceivedOp[[i]]$BLADDEROP, DemoList[[i]], drop = T)
}

xFromNA <- function(item, x) {
if (is.na(item)) {
item <- x
}
item
}

for (i in 1:9) { ## Orders INC_KEYs for those who mortality data according to whether they received an operation or not, the outside loop goes by year.
for (j in 1:nrow(ReceivedOp[[i]])) { ## The inside loop goes by INC_KEY from the ReceivedOp list
if (sum(DISCHARGE[[i]]$DECEASED[DISCHARGE[[i]]$INC_KEY == ReceivedOp[[i+5]]$INC_KEY[j]], na.rm = T) > 0) {
OrganizedDischarge[[i+5]][j] <- TRUE
}
else if ((xFromNA(sum(DISCHARGE[[i]]$DECEASED[DISCHARGE[[i]]$INC_KEY == ReceivedOp[[i+5]]$INC_KEY[j]]), -1) == 0)) {
OrganizedDischarge[[i+5]][j] <- FALSE
}
}
}

for (i in 1:9) { ## What age group, whether they had an EP or IP rupture, and whether they died.
AgexRuptureTypexMortalityInteractions[[i]] <- interaction(DemoList[[i+5]], RuptureTypeList[[i+5]], OrganizedDischarge[[i+5]])
}

for (i in 1:9) {
AgexRupturexOpxMort[[i]] <- interaction(OrganizedDischarge[[i+5]], AgexRuptureTypexOperationTypeInteractions[[i+5]])
}

pelvicDCODES <- c(808.0, 808.1, 808.2, 808.3, 808.4, 808.41, 808.42, 808.43, 808.49, 808.5, 808.51, 808.52, 808.59, 808.8, 808.9)

for (i in 1:14) { ## Here we are getting whether it is an EP or IP rupture, by converting to numeric before checking, we remove the difference between 867 and 867.0.
for (j in 1:nrow(ReceivedOp[[i]])) { ## Going through by INC_KEY in the ReceivedOp
for (k in 1:16) {
if (pelvicDCODES[k] %in% suppressWarnings(as.numeric(as.character(D_CODES[[i]]$DCODE[D_CODES[[i]]$INC_KEY == ReceivedOp[[i]]$INC_KEY[j]])))) { ## Try to see what happens if you mess around with this condition in a separate file so you understand why it is the way it is.
pelvicFractures[[i]][j, k] <- TRUE
}
else {
pelvicFractures[[i]][j, k] <- FALSE
}
}
}
}

pelvisInteractions <- rep(list(rep(list(), 14)), 15)

pelvicDCODES <- as.character(pelvicDCODES)

for (j in 1:14) { ## Pre-allocate memory
RuptureType <- rep(NA, nrow(ReceivedOp[[j]]))
for (i in 1:15) {
pelvisInteractions[[i]][[j]] <- RuptureType
}
}

names(pelvisInteractions) <- pelvicDCODES

for (i in 1:15) {
for (j in 1:9) {
pelvisInteractions[[i]][[j+5]] <- interaction(AgexRuptureTypexOperationTypeInteractions[[j+5]], as.logical(pelvicFractures[[j+5]][ , i]))
}
}

pelvisInteractions <- lapply(pelvisInteractions, function(x) 'names<-'(x, 2002:2015))

pelvisInteractions <- rapply(pelvisInteractions, table, how = "replace")

# 3. Collect counts

EPvIP <- t(data.frame(lapply(RuptureTypeList, table)))[-(1+(1:13)*2), ] ## Calculate frequencies by year, same for all the rest
colnames(EPvIP) <- EPvIP[1, ]
EPvIP <- EPvIP[-1, ]
rownames(EPvIP) <- 2002:2015 ## Put into nice format for writing to file, same for all the rest

ReceivedOperationAndRuptureType <- t(data.frame(lapply(InteractionTerms, table)))[-(1+(1:13)*2), ]
colnames(ReceivedOperationAndRuptureType) <- ReceivedOperationAndRuptureType[1, ]
ReceivedOperationAndRuptureType <- ReceivedOperationAndRuptureType[-1, ]
rownames(ReceivedOperationAndRuptureType) <- 2002:2015

ReceivedOperationAndRuptureTypeAndAge <- t(data.frame(lapply(AgexRuptureTypexOperationTypeInteractions, table)))[-(1+(1:13)*2), ]
colnames(ReceivedOperationAndRuptureTypeAndAge) <- ReceivedOperationAndRuptureTypeAndAge[1, ]
ReceivedOperationAndRuptureTypeAndAge <- ReceivedOperationAndRuptureTypeAndAge[-1, ]
rownames(ReceivedOperationAndRuptureTypeAndAge) <- 2002:2015

ReceivedOperationAndAge <- t(data.frame(lapply(AgexOperationTypeInteractions, table)))[-(1+1:13*2), ]
colnames(ReceivedOperationAndAge) <- ReceivedOperationAndAge[1, ]
ReceivedOperationAndAge <- ReceivedOperationAndAge[-1, ]
rownames(ReceivedOperationAndAge) <- 2002:2015

AgeAndRuptureTypeAndMortality <- t(data.frame(lapply(AgexRuptureTypexMortalityInteractions[1:9], table)))[-(1+1:9*2), ]
colnames(AgeAndRuptureTypeAndMortality) <- AgeAndRuptureTypeAndMortality[1, ]
AgeAndRuptureTypeAndMortality <- AgeAndRuptureTypeAndMortality[-1, ]
rownames(AgeAndRuptureTypeAndMortality) <- 2007:2015

Mortality <- t(data.frame(lapply(OrganizedDischarge[6:14], table)))[-(1+1:9*2), ]
colnames(Mortality) <- Mortality[1, ]
Mortality <- Mortality[-1, ]
rownames(Mortality) <- 2007:2015

ReceivedAgexRupturexOpxMort <- t(data.frame(lapply(AgexRupturexOpxMort[1:9], table)))[-(1+1:9*2), ]
colnames(ReceivedAgexRupturexOpxMort) <- ReceivedAgexRupturexOpxMort[1, ]
ReceivedAgexRupturexOpxMort <- ReceivedAgexRupturexOpxMort[-1, ]
rownames(ReceivedAgexRupturexOpxMort) <- 2007:2015

pelvisInteractions <- lapply(pelvisInteractions, function(x) x[-(1:5)])

for (i in 1:length(pelvisInteractions)) {
for (j in 1:length(pelvisInteractions[[i]])) {
pelvisResults <- t(data.frame(pelvisInteractions[[i]]))[-(1+(1:13)*2), ]
colnames(pelvisResults) <- pelvisResults[1, ]
pelvisResults <- pelvisResults[-1, ]
rownames(pelvisResults) <- 2007:2015
}
write.csv(pelvisResults, paste0("pelvisdcode", names(pelvisInteractions)[i], "xEverything.csv"))
}

# 4. Write to files

write.csv(EPvIP, "resultsEPvIP.csv")
write.csv(ReceivedOperationAndRuptureType, "resultsOperationAndRupture.csv")
write.csv(ReceivedOperationAndRuptureTypeAndAge, "resultsOperationAndRuptureAndAge.csv")
write.csv(ReceivedOperationAndAge, "resultsOperationAndAge.csv")
write.csv(Mortality, "resultsMortality.csv")
write.csv(AgeAndRuptureTypeAndMortality, "resultsAgeRuptureTypeMortality.csv")
write.csv(ReceivedAgexRupturexOpxMort, "resultsOperationAndRuptureAndAgeAndMortality.csv")

His request? Help me understand what this code does, because I don’t know how it got from the source data to the output data.

He had two obstacles to overcome that made mystification inevitable. He had no coding experience in the imperative/procedural style of C, C++, Java, Python and most other languages, and he was new to R. Now, he was reading code in R, a functional language, written primarily in an imperative style over 233 lines.

I thought that a more idiomatic version would clarify the data transformation. We talked of which csv files were of interest, what variables were important, what the output should look like and what the plans were for analyzing the data.

As a first cut, I returned 72 lines of more idiomatic R in the tidyverse style to look at a single year of the data.



#import libraries

library(tidyverse)
setwd("/Users/rc/projects/working directory/RDS AY 2015") 

#set print display options

options(pillar.sigfig = 10) #control display of tibble

#Get patient demographics

patients <- as.tibble(read.csv("RDS_DEMO.csv", stringsAsFactors = FALSE))
patients <- patients %>% select(INC_KEY, AGE, GENDER)
patients <- patients %>% select(INC_KEY, AGE, GENDER) %>% mutate(ADULT = ifelse(AGE > 17, "TRUE", "FALSE"))
 
#eliminate duplicates
patients <- distinct(patients)

#Get discharge status

discharges <- as.tibble(read.csv("RDS_DISCHARGE.csv", stringsAsFactors = FALSE))
discharges <- discharges %>% select(INC_KEY,HOSPDISP) %>%  filter(HOSPDISP != "Not Applicable BIU 1" & HOSPDISP != "Not Known/Not Recorded BIU 2") %>% mutate(Expired = ifelse(HOSPDISP == "Deceased/Expired", "TRUE", "FALSE")) %>% select(-HOSPDISP)

#Get intervention codes

p_codes <- as.tibble(read.csv("RDS_PCODE.csv", stringsAsFactors = FALSE))
p_codes <- p_codes %>% select(INC_KEY, PCODE)
pelvic <- p_codes %>% filter(PCODE >= 57.6 & PCODE <= 57.89 | PCODE == 57.93) %>% mutate(PCODE = as.logical(PCODE))
colnames(pelvic) <- c("INC_KEY","PELVIC") # 57.93 56.6:57.89 & 57.93

#Diagnostic codes

d_codes <- as.tibble(read.csv("RDS_DCODE.csv", stringsAsFactors = FALSE))
d_codes <- d_codes %>% select(INC_KEY, DCODE)
Rupture <- d_codes %>% filter(DCODE > 866.99 & DCODE < 867.2)

#Join patients, discharge, then patients, treatment, then T/F pelvic then DCODES, add year and reorder

patients <- inner_join(patients, discharges, by = "INC_KEY")

#Treatment codes

#ID patients with no pelvic procedure

pelvic_false <- setdiff(patients$INC_KEY, pelvic$INC_KEY)

#Add field for PELVIC t/f

patients <- patients %>% mutate(PELVIC = ifelse(INC_KEY %in% pelvic_false, "FALSE", "TRUE")) %>% mutate(PELVIC = as.logical(PELVIC))

#Add fields for 867 series

patients <- inner_join(patients, Rupture, by = "INC_KEY")

#Add year

patients <- patients %>% mutate(YEAR = 2015)

#Rename columns

colnames(patients) <- c("INC_KEY", "Age", "Sex", "Adult", "Expired", "Intervention", "Rupture", "Year")

#Change Expired to logical

patients <- patients %>% mutate(Expired = as.logical(Expired))

#Censor anomolies

patients <- patients %>% filter(Age >= 0)

#SAVE

write.csv(patients, "../combined/patients_2015.csv")

That was something he was able to understand, and we have been working together to refine, possible because we now share a mutually intelligible language.

I should emphasize that I have no criticism of the programmer, the style of programming or the process that left the user in the dark. The lesson for everyone in the process of converting facts to data to data sets to programming objects is that mutual intelligibility is key.

We have run into some challenges that we are working to correct, inconsistent column naming conventions, data coding and similar routine issues. The major problem was the underlying data structure in which the same unit of observation, a patient, had multiple observations in multiple files, classic untidy data. Pulling all of the source data into a single data structure profilerated duplicate patient records that required a method to collapse into a single row.

That proved challenging for a surprising reason¹, about which more when the tentative solution has been more thoroughly tested.

The tibble, as opposed to its constituent vector columns is difficult to iterate over as a whole when varying numbers of rows interfere with the expectations of the standard tools to apply functions to vectors.↩︎

Less coding and more analysis

Sun, 09 Dec 2018 00:00:00 +0000

I urge you to install.packages("ExPanDaR") and then try the following:


library(ExPanDaR)
wb <- read.csv("https://joachim-gassen.github.io/data/wb_condensed.csv")
ExPanD(wb, cs_id = "country", ts_id = "year")

from Interactive panel EDA with 3 lines of code, which doesn’t show the output to its best advantage, for which see online demo

Color Map Atlas for Continuously Scaled Maps

Mon, 03 Dec 2018 00:00:00 +0000

## Please note: Alaska and Hawaii are being shifted and are not to scale.

Color, Design and Communication

Graphic design standards, especially for color, are an official Good Thing. Branding, visual cueing, consistency. Map colors are hard, especially when dealing with continuous data.

An example of continuous data

The U.S. Bureau of Economic Analysis publishes quarterly statistics on gross domestic product by state.¹. For the second quarter 2018, it provided the following.

(#tab:gdp_table)2018 Second Quarter GDP for the 50 States
NAME	GDP	geometry
Arizona	322,995	MULTIPOLYGON (((-1111066 -8…
Arkansas	123,605	MULTIPOLYGON (((557903.1 -1…
California	2,791,626	MULTIPOLYGON (((-1853480 -9…
Colorado	343,316	MULTIPOLYGON (((-613452.9 -…
Connecticut	264,949	MULTIPOLYGON (((2226838 519…
Georgia	557,294	MULTIPOLYGON (((1379893 -98…
Illinois	820,408	MULTIPOLYGON (((868942.5 -2…
Indiana	348,366	MULTIPOLYGON (((1279733 -39…
Louisiana	236,514	MULTIPOLYGON (((1080885 -16…
Minnesota	351,868	MULTIPOLYGON (((594664.5 21…
Mississippi	109,181	MULTIPOLYGON (((1052956 -15…
Montana	47,040	MULTIPOLYGON (((-864991.3 5…
New Mexico	93,323	MULTIPOLYGON (((-568680.6 -…
North Dakota	51,972	MULTIPOLYGON (((-39585.46 1…
Oklahoma	187,043	MULTIPOLYGON (((-34.95064 -…
Pennsylvania	746,635	MULTIPOLYGON (((1734901 -36…
Tennessee	345,710	MULTIPOLYGON (((1467705 -78…
Virginia	507,407	MULTIPOLYGON (((2133159 -45…
Delaware	71,164	MULTIPOLYGON (((2058738 -29…
West Virginia	72,288	MULTIPOLYGON (((1827955 -42…
Wisconsin	319,280	MULTIPOLYGON (((721464.2 26…
Wyoming	37,644	MULTIPOLYGON (((-324546.2 -…
Alabama	210,298	MULTIPOLYGON (((1145349 -15…
Florida	971,078	MULTIPOLYGON (((1994020 -19…
Idaho	72,065	MULTIPOLYGON (((-873311.5 1…
Kansas	159,167	MULTIPOLYGON (((40850.31 -8…
Maryland	396,750	MULTIPOLYGON (((2063382 -46…
New Jersey	597,752	MULTIPOLYGON (((2096848 -22…
North Carolina	534,880	MULTIPOLYGON (((2173840 -77…
South Carolina	221,061	MULTIPOLYGON (((1607185 -91…
Washington	518,701	MULTIPOLYGON (((-1642369 55…
Vermont	32,490	MULTIPOLYGON (((2197053 124…
Utah	163,887	MULTIPOLYGON (((-909692.9 -…
Iowa	183,496	MULTIPOLYGON (((717165 -185…
Kentucky	199,833	MULTIPOLYGON (((934668.1 -8…
Maine	61,230	MULTIPOLYGON (((2391419 325…
Massachusetts	537,860	MULTIPOLYGON (((2364278 491…
Michigan	503,384	MULTIPOLYGON (((1112263 184…
Missouri	303,577	MULTIPOLYGON (((936191 -904…
Nebraska	118,536	MULTIPOLYGON (((-336456.4 -…
Nevada	155,890	MULTIPOLYGON (((-1188405 -4…
New Hampshire	80,427	MULTIPOLYGON (((2305592 212…
New York	1,583,899	MULTIPOLYGON (((2160781 -11…
Ohio	641,630	MULTIPOLYGON (((1425651 -22…
Oregon	224,764	MULTIPOLYGON (((-1676653 30…
Rhode Island	59,201	MULTIPOLYGON (((2325633 198…
South Dakota	49,706	MULTIPOLYGON (((-325003.8 -…
Texas	1,640,409	MULTIPOLYGON (((314225.2 -1…
Alaska	51,214	MULTIPOLYGON (((-1434617 -2…
Hawaii	88,593	MULTIPOLYGON (((-231495.6 -…

The two obvious things from the chart are that the numbers differ and that in general the larger states in population are also the larger states in GDP. Whenever making maps of continuous data, they will almost always turn out like this, unless there are marked regional differences and you take the default option.

Here is the GDP data, also with the default option.

Three Problems: One Major, One Important and One Nuisance

Major: The GDP map doesn’t tell you anything you could not have guessed from the population map
You can see subtle differences in shade, but there are too many, and a small portion of the population is blue colorblind.
Nuisance: Can’t see the state names against the dark background

The Major Problem: Counts Need Transformation

If we are looking for more interesting differences among states, we should put them on a common footing, so that population size alone doesn’t dominate the effects. For disease, as an example, cases per hundred thousand is commonly used. Here’s what the GDP data looks like on that basis.

The Important Problem

The default continuous blue scale, even with transformed data, still requires squinting to distinguish different values.

Alpha adjustment

We can tinker with that by applying an alpha (density) filter.

The signature for scale_alpha_continuous is

scale_alpha_continuous(…, range = c(0.1, 1))

The range can be narrowed

scale_alpha_continuous(range = c(0.1, 0.75))

or even further lightened

scale_alpha_continuous(range = c(0.1, 0.50))

Beyond Blue Palette Solutions

Viridis palette

Five bright (some might say garish) scales are available.

scale_fill_viridis_c(option="magma")

scale_fill_viridis_c(option="inferno")

scale_fill_viridis_c(option="plasma")

scale_fill_viridis_c(option="viridis")

Fortunately, the palette can be muted with an alpha density filter

scale_fill_viridis_c(alpha = 0.50, option="cividis")

ggplot’s `scale_fill_gradient` can also over-ride default colors

The signature of scale_fill_gradient is

scale_fill_gradient(..., low = "#132B43", high = "#56B1F7", space = "Lab", na.value = "grey50", guide = "colourbar", aesthetics = "fill")

in the blue range. You can adjust that with the arguments to low and high. The following example uses #FBF7F6 and #A97263.

How did I come up with those particular values?

library(munsell)

show_col(seq_gradient_pal("white", mnsl("10R 4/6"))(x))(seq(0, 1, length.out = 25)))

This approach, and other do-it-yourself color palette selection approaches can become a time sink.See R color cheatsheet A better approach is to have a selection of pre-made color cards, and try them out until one of them highlights the differences you are trying to show.

RColorBrewer

This set of palettes plays well with ggplot and should answer most needs.

RColorBrewer

The three sections are for sequential (i.e., continuous), divergent (a neutral color in the center progressing toward more intense colors in both direction) or qualitative, for discrete data but coercible into continuous data.

Through ggplot’s scale_colour_distiller function, any of these can be smoothly interpolated into a six-interval scale.

All of the scales can be flipped, working from left-to-right on the palette or from right-to-left. Direction is indicated by direction = -1 or direction = 1.

Color abbreviations:

Yl = Yellow
Or = Orange
Rd = Red
Br = Brown
Gn = Green
Pu = Purple
Bu = Blue

There are also:

Reds
Oranges
Greys
Greens
Blues

The signature for scale_colour_distiller is

scale_fill_distiller(type = “seq”, palette = “YlOrRd”, direction = -1, labels = comma, guide = “colourbar”, aesthetics = “fill”)

Palette Snippets

Usage ggplot_map_object + [name of color scale]



    Alpha_default <- scale_alpha_continuous(range = c(0.1, 1))
  
    Alpha_75      <- scale_alpha_continuous(range = c(0.1, 0.75))
  
    Alpha_50      <- scale_alpha_continuous(range = c(0.1, 0.50))

    magma         <- scale_fill_viridis_c(option="magma")
  
    inferno       <- scale_fill_viridis_c(option="inferno")
  
    plasma        <- scale_fill_viridis_c(option="plasma")
  
    viridis       <- scale_fill_viridis_c(option="viridis")
  
    cividis       <- scale_fill_viridis_c(option="cividis")

    munsell_brown <- scale_fill_gradient(low = "#FBF7F6", high =  "#A97263", space = "Lab", na.value = "grey50", guide = "colourbar", aesthetics = "fill")

    # RColorBrewer

    BluesNeg1    <- scale_fill_distiller(type = "seq", palette = "Blues", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BluesPos1    <- scale_fill_distiller(type = "seq", palette = "Blues", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BuGnNeg1    <- scale_fill_distiller(type = "seq", palette = "BuGn", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BuGnPos1    <- scale_fill_distiller(type = "seq", palette = "BuGn", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BuPuNeg1    <- scale_fill_distiller(type = "seq", palette = "BuPu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BuPuPos1    <- scale_fill_distiller(type = "seq", palette = "BuPu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GnBuNeg1    <- scale_fill_distiller(type = "seq", palette = "GnBu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GnBuPos1    <- scale_fill_distiller(type = "seq", palette = "GnBu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GreensNeg1    <- scale_fill_distiller(type = "seq", palette = "Greens", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GreensPos1    <- scale_fill_distiller(type = "seq", palette = "Greens", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GreysNeg1    <- scale_fill_distiller(type = "seq", palette = "Greys", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    GreysPos1    <- scale_fill_distiller(type = "seq", palette = "Greys", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    OrangesNeg1    <- scale_fill_distiller(type = "seq", palette = "Oranges", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    OrangesPos1    <- scale_fill_distiller(type = "seq", palette = "Oranges", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    OrRdNeg1    <- scale_fill_distiller(type = "seq", palette = "OrRd", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    OrRdPos1    <- scale_fill_distiller(type = "seq", palette = "OrRd", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuBuGnNeg1    <- scale_fill_distiller(type = "seq", palette = "PuBuGn", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuBuGnPos1    <- scale_fill_distiller(type = "seq", palette = "PuBuGn", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuBuNeg1    <- scale_fill_distiller(type = "seq", palette = "PuBu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuBuPos1    <- scale_fill_distiller(type = "seq", palette = "PuBu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuRdNeg1    <- scale_fill_distiller(type = "seq", palette = "PuRd", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuRdPos1    <- scale_fill_distiller(type = "seq", palette = "PuRd", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PurplesNeg1    <- scale_fill_distiller(type = "seq", palette = "Purples", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PurplesPos1    <- scale_fill_distiller(type = "seq", palette = "Purples", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdPuNeg1    <- scale_fill_distiller(type = "seq", palette = "RdPu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdPuPos1    <- scale_fill_distiller(type = "seq", palette = "RdPu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RedsNeg1    <- scale_fill_distiller(type = "seq", palette = "Reds", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RedsPos1    <- scale_fill_distiller(type = "seq", palette = "Reds", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlGnBuNeg1    <- scale_fill_distiller(type = "seq", palette = "YlGnBu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlGnBuPos1    <- scale_fill_distiller(type = "seq", palette = "YlGnBu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlGnNeg1    <- scale_fill_distiller(type = "seq", palette = "YlGn", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlGnPos1    <- scale_fill_distiller(type = "seq", palette = "YlGn", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlOrBrNeg1    <- scale_fill_distiller(type = "seq", palette = "YlOrBr", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlOrBrPos1    <- scale_fill_distiller(type = "seq", palette = "YlOrBr", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlOrRdNeg1    <- scale_fill_distiller(type = "seq", palette = "YlOrRd", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    YlOrRdPos1    <- scale_fill_distiller(type = "seq", palette = "YlOrRd", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BrBGNeg1    <- scale_fill_distiller(type = "seq", palette = "BrBG", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    BrBGPos1    <- scale_fill_distiller(type = "seq", palette = "BrBG", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PiYGNeg1    <- scale_fill_distiller(type = "seq", palette = "PiYG", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PiYGPos1    <- scale_fill_distiller(type = "seq", palette = "PiYG", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PRGnNeg1    <- scale_fill_distiller(type = "seq", palette = "PRGn", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PRGnPos1    <- scale_fill_distiller(type = "seq", palette = "PRGn", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuOrNeg1    <- scale_fill_distiller(type = "seq", palette = "PuOr", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PuOrPos1    <- scale_fill_distiller(type = "seq", palette = "PuOr", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdBuNeg1    <- scale_fill_distiller(type = "seq", palette = "RdBu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdBuPos1    <- scale_fill_distiller(type = "seq", palette = "RdBu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdGyNeg1    <- scale_fill_distiller(type = "seq", palette = "RdGy", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdGyPos1    <- scale_fill_distiller(type = "seq", palette = "RdGy", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdYlBuNeg1    <- scale_fill_distiller(type = "seq", palette = "RdYlBu", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdYlBuPos1    <- scale_fill_distiller(type = "seq", palette = "RdYlBu", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdYlGnNeg1    <- scale_fill_distiller(type = "seq", palette = "RdYlGn", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    RdYlGnPos1    <- scale_fill_distiller(type = "seq", palette = "RdYlGn", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    SpectralNeg1    <- scale_fill_distiller(type = "seq", palette = "Spectral", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    SpectralPos1    <- scale_fill_distiller(type = "seq", palette = "Spectral", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    AccentNeg1    <- scale_fill_distiller(type = "seq", palette = "Accent", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    AccentPos1    <- scale_fill_distiller(type = "seq", palette = "Accent", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Dark2Neg1    <- scale_fill_distiller(type = "seq", palette = "Dark2", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Dark2Pos1    <- scale_fill_distiller(type = "seq", palette = "Dark2", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PairedNeg1    <- scale_fill_distiller(type = "seq", palette = "Paired", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    PairedPos1    <- scale_fill_distiller(type = "seq", palette = "Paired", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Pastel1Neg1    <- scale_fill_distiller(type = "seq", palette = "Pastel1", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Pastel1Pos1    <- scale_fill_distiller(type = "seq", palette = "Pastel1", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Pastel2Neg1    <- scale_fill_distiller(type = "seq", palette = "Pastel2", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Pastel2Pos1    <- scale_fill_distiller(type = "seq", palette = "Pastel2", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set1Neg1    <- scale_fill_distiller(type = "seq", palette = "Set1", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set1Pos1    <- scale_fill_distiller(type = "seq", palette = "Set1", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set2Neg1    <- scale_fill_distiller(type = "seq", palette = "Set2", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set2Pos1    <- scale_fill_distiller(type = "seq", palette = "Set2", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set3Neg1    <- scale_fill_distiller(type = "seq", palette = "Set3", direction = -1, labels = comma,  guide = "colourbar", aesthetics = "fill")

    Set3Pos1    <- scale_fill_distiller(type = "seq", palette = "Set3", direction = 1, labels = comma,  guide = "colourbar", aesthetics = "fill")

Credits

R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.
Simon Urbanek and Jeffrey Horner (2015). Cairo: R graphics device using cairo graphics library for creating high-quality bitmap (PNG, JPEG, TIFF), vector (PDF, SVG, PostScript) and display (X11 and Win32) output. R package version 1.5-9. https://CRAN.R-project.org/package=Cairo
H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016
Hao Zhu (2018). kableExtra: Construct Complex Table with ‘kable’ and Pipe Syntax. R package version 0.9.0. https://CRAN.R-project.org/package=kableExtra
Yihui Xie (2018). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.20. Yihui Xie (2015) Dynamic Documents with R and knitr. 2nd edition. Chapman and Hall/CRC. ISBN 978-1498716963 Yihui Xie (2014) knitr: A Comprehensive Tool for Reproducible Research in R. In Victoria 6. Stodden, Friedrich Leisch and Roger D. Peng, editors, Implementing Reproducible Computational Research. Chapman and Hall/CRC. ISBN 978-1466561595
Charlotte Wickham (2018). munsell: Utilities for Using Munsell Colours. R package version 0.5.0. https://CRAN.R-project.org/package=munsell
Erich Neuwirth (2014). RColorBrewer: ColorBrewer Palettes. R package version 1.1-2. https://CRAN.R-project.org/package=RColorBrewer
Hadley Wickham (2018). scales: Scale Functions for Visualization. R package version 1.0.0.https://CRAN.R-project.org/package=scales
Pebesma, E., 2018. Simple Features for R: Standardized Support for Spatial Vector Data. The R Journal, https://journal.r-project.org/archive/2018/RJ-2018-009/
Kyle Walker (2018). tidycensus: Load US Census Boundary and Attribute Data as ‘tidyverse’ and22 ‘sf’-Ready Data Frames. R package version 0.8.1. https://CRAN.R-project.org/package=tidycensus
Hadley Wickham (2017). tidyverse: Easily Install and Load the ‘Tidyverse’. R package version 1.2.1. https://CRAN.R-project.org/package=tidyverse

“Source: Bureau of Economic Analysis, Gross Domestic Product By State, 2nd Quarter 2018 https://goo.gl/Jc6XyK”↩︎

Commercial data science: a parable of downsides of crisp deployment

Thu, 08 Nov 2018 00:00:00 +0000

Narcocorridos are a Mexican musical genre celebrating the drug trafficking culture. As you can read in wiki, it has a surprisingly long history and is popular despite official attempts to suppress it.

Stay with me, there’s a parable here about R, Python, DevOps, Bias Toward Action and leaving change on the table, in two senses.

A mild English flavor of a narcocorido is Townes van Zandt’s ballad Pancho and Lefty, his best known work in the pre-pop Country genre. It strips the Spanish version to its essentials. It was made popular by Emmy Lou Harris and the duet by Willie Nelson and Merle Haggard. It may not be your cup of tea musically, but the story line provides a hook to my thoughts about what we used to call R&D.

Here are the essential passages:

Pancho was a bandit boy
His horse was fast as polished steel
He wore his gun outside his pants
For all the honest world to fear
Pancho met his match you know
On the deserts down in Mexico
Nobody heard his dying words
But that’s the way it goes

All the Federales say
They could’ve had him any day
They only let him slip away
Out of kindness I suppose

Lefty, he can’t sing the blues
All night long like he used to
The dust that Pancho bit down south
Ended up in Lefty’s mouth
The day they laid poor Pancho low
Lefty split for Ohio
Where he got the bread to go
There ain’t nobody knows

The poets tell how Pancho fell
And Lefty’s living in a cheap hotel
The desert’s quiet and Cleveland’s cold
And so the story ends we’re told
Pancho needs your prayers it’s true
But save a few for Lefty too
He only did what he had to do
And now he’s growing old

So, here’s a reading. Pancho is an outlaw. He has some relation to Lefty. The authorities tolerated him, but finally killed Pancho, presumably on a tip from Lefty, who, it is implied was paid. Pancho achieves a minor version of Homeric immortality and Lefty will be forgotten.

Let’s translate this to the contemporary world of applied statistics, which we call data science. As we all know, much of it is in the service of large organizations who are anxious to use the methods in pursuit of profits. This is as we have designed the knowledge economy.

Next, picture Pancho and Lefty as stereotypical foot soldiers in the war to deploy this new magic for the benefit of the Federales, who represent that elusive, shifting group management.

Lefty is what used to be called a company man. Point and shoot. When he finds a solution that conforms to the designated problem, you can count on him to bring it online fast.

Pancho, on the other hand, is a rebel without a cause beyond his own curiosity, oblivious to the social niceties of an organization with any degree of hierarchy, but is wildly creative. He came up with all known conventional solutions, totally novel solutions, solutions to problems no one yet recognized. Lefty picked one of the conventional solutions.

Pancho is the R in R&D and Lefty is the D. One might speculate that Pancho dreams in R and Lefty in Python, while muttering complaints about how much easier it would be to translate prototypes from an interpreted procedural language than to have to reinvent the translation from a functional language every damned time. Pancho could care less.

Comes a time when management gathers up all of Pancho’s reviews, see how smart he is, but he’s just not a team player or very productive. In fact, he’s a distraction with all his crazy irrelevant and impractical ideas. His costs could better be deployed elsewhere.

Pancho doesn’t have to be a genius, he just has to produce hidden gems for purposes of my tale. Manufacturing jewelers get free janitorial services from outfits that recover the gold dust.

But no one is looking at Pancho’s failures and debris for the germ of innovation. In the completion of Rumsfeld’s typology of knowledge, Pancho provides unknown knowns. Knowns because they exist under everyone’s nose. Unknowns because no one pauses to identify the scent.

AT&T and IBM in their past incarnations knew how to find, sort and exploit the best of the happy byproducts of R&D, and prospered with innovative technologies.

And Lefty? Why does he end up in the ash heap? He did a very good job at creating a commodity. In the absence of monopoly or oligarchy the marginal values of commodities inevitably approach zero. The only way to recover from becoming a commodity to becoming a resource is the emergence of an innovative technology that transforms the commodity into an entirely new technology.

It’s an old canard: Learn from your successes, not your failures because you can always invent new ways of screwing up. And virtually all post-mortem lessons learned are focused on avoiding screwing up in the same way next time or more cheaply replicating the next success.

A few, somewhere, must look at their failures as mine tailings that might contain unpicked gems. They get the germ of innovation that the others miss.

R and Haskell, meant for each other?

Wed, 07 Nov 2018 00:00:00 +0000

R has a notoriously steep learning curve. I once moaned that help() needed it’s own help(help), which it does, but did nothing to help me understand the paradigm of function signatures. And why were control loops like for so frowned upon. All sorts of great functionality, but how do you, like program it.

As I progressed in my ability to overcome my underlying lack of grok, I took a dip in Haskell, and discovered the concept of a functional language, essentially everything is $f(x) = y$ as opposed to an imperative/procedural language – first do this, then do that, then perform some operation on both of them, and then, and then … .

A lonely neuron potentiated its axon and discharged a signal that tapped into several related networks that did their hidden layer human intelligence thing and agreed: R is a functional language, like Haskell.

Suddenly I learned to take R function signatures much, much more seriously, along with linear alegbra type operations as the efficient substitute for control loops.

R being an interpreted language is sometimes called upon to perform calculations that push its available memory and speed past the point of usefulness. For that, we have the ability to plug in compiled modules, to expose their functionality to the interpreter. See rccp

Another lonely neuron fired today. In scaling many R scripts to large-production scale, at least outside of the RShiny Server gui context, the script, its packages, dependences, functions, transformations and logic have to be reverse engineered into C++ or another compiled language. That involves the trailing sea anchor of translating a functional language into an imperative language. And, if you’re conscientious, building and maintaining libraries for the frequently used algorithms so you don’t have to start from scratch everytime, documenting them in native fashion while also being R friendly to enable the original developers to follow along.

Why not do this in Haskell a compiled functional language? Write an “Rh” package similar to Rcpp and give back to the R author as a hook to a faster implementation of standard R? In time, you would have a domain specific dialect of R that was essentially compiled. Haskell is also said (above my pay grade) to have features of Curry-Howard correspondence, a sort of self-proving program paradigm with the potential, as I understand it, to minimize the possibilities of logical errors and the stated aim of precluding side effects.

#rstats #haskell

n?

Sat, 27 Oct 2018 00:00:00 +0000

Got n?

There was a 100% increase in 1-hour parking meter costs this year!

(Imagined conversation in a small town.)

It appears to be hardwired into our wetware to forget to look for the underlying integers. You might ask, for example from what to what? and learn that it went up from a nickle to a dime.

This is a manifestation what Daniel Kahneman, author of Thinking Fast, and Slow (2011, Ferrar, Straus and Giroux, New York) calls The Law of Small Numbers. Our favorite ballplayer has a hot hand (random variation), an image of a religious figure appears in the bird tracks on a beach (randomness doesn’t look uniform, there are all sorts of random patterns) and, the big tell, statistical findings based on a small sample size, n are particularly vulnerable to random effects.

In one example the author gives as an example a survey of 1,662 schools in Pennsylvania. Of the top 50, six were small schools, about four times as many as you would expect to be exceptional. So, a well-known philanthropist, joined by some famous foundations, set up a program to split larger schools up into smaller schools. If small schools do better than average, after all, it only makes sense.

Unfortunately, turning the question around, small schools also tend to have worse outcomes than average. The smaller schools, good, bad or indifferent, are simply more variable.

Look for n, your first line of defense against putting your money where the facts aren’t.

Month arithmetic in R, a quick guide

Sat, 20 Oct 2018 00:00:00 +0000

Here are questions you might want to ask about months:

How many more months until September 1? months_to
How many more calendar months until September 1? cal_months_to
How many months between this month and September 1? months_between
How many more calendar months between this month and September 1? cal_months_between
In how many calendar months must I wait until September 1? months_inclusive

Assume we are asking the question on February 1 of a non-leap year.

Don’t worry; I’m not going to give a tedious account of all the potential ambiguities and subtle points of calculation. I just want to focus on the last question, which is common in the everyday worlds of finance and accounting, and point out that many software functions allow you to calculate months_between, but leave you hanging for months_inclusive.

Here are the months: February, March, April, May, June, July, August, September, eight different months of which six are not February or September.

My motivation is in knowing of payments due on the first of each month, what percentage were paid. Because payments were due on each of the eight months, I need not months_between, but months_inclusive, and not only half inclusive – I want to include both February and September.

What I found is a cautionary tale of embedded assumptions, some of which are made explicit, some of which are implied and others of which are opaque. Let’s look at how various months_between functions work in some popular environments.

First, Excel.

Excel example

This has to fall in the opaque category: semi-inclusive with two date pairs and non-inclusive with one.

Python

IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %paste
from datetime import datetime

def diff_month(d1, d2):
    return (d1.year - d2.year) * 12 + d1.month - d2.month

d1 = datetime(2018,2,1,0,0)

d2 = datetime(2017,8,1,0,0)
## -- End pasted text --

In [2]: diff_month(d1,d2)
Out[2]: 6

Credit: John La Rooy https://goo.gl/PhJeKb

This implementation, which was the simplest I found, but consistent with the others I looked at, is a straight forward months_between.

MySQL

MariaDB [dlf]> create table temp (
    -> id int not null,
    -> start date,
    -> end date
    -> );
Query OK, 0 rows affected (0.011 sec)

MariaDB [dlf]>
MariaDB [dlf]> INSERT INTO temp (id, start, end) VALUES (1, '2018-02-01', '2017-09-01');
Query OK, 1 row affected (0.002 sec)

MariaDB [dlf]>
MariaDB [dlf]> select month(end) - month(start) as 'Elapsed' from temp;
+---------+
| Elapsed |
+---------+
|       7 |
+---------+
1 row in set (0.000 sec)

MariaDB [dlf]> select TIMESTAMPDIFF(MONTH, start, end) as "Elapsed" from temp;
+---------+
| Elapsed |
+---------+
|      -5 |
+---------+
1 row in set (0.000 sec)

probably because I used DATE objects instead of DATETIME objects. (The lost month arises from conventions of time of day.)

R

Now that we’ve seen that the fully-inclusive calculation is rare, I have the sad duty to report that it’s a problem in R as well.

Base

I started my search in the {base} package by looking at the documentation for difftime. The tell was in the function signature

difftime(time1, time2, tz,
     units = c("auto", "secs", "mins", "hours",
               "days", "weeks"))

(No units for months.)

lubridate

Next, I tried the lubridate package ¹, and it seemed to provide a way forward. From the documentation:

Adding months frustrates basic arithmetic because consecutive months have different lengths. With other elements, it is helpful for arithmetic to perform automatic roll over. For example, 12:00:00 + 61 seconds becomes 12:01:01. However, people often prefer that this behavior NOT occur with months. For example, we sometimes want January 31 + 1 month = February 28 and not March 3. %m+% performs this type of arithmetic. Date %m+% months(n) always returns a date in the nth month after Date. If the new date would usually spill over into the n + 1th month, %m+% will return the last day of the nth month (rollback()). Date %m-% months(n) always returns a date in the nth month before Date.

However, the arithmetic calculations required datetime objects (dates, plus times of day) required arguments in the %m-% operators to be of different classes. During several hours of tinkering, it seemed close to doing what I was looking for, but perhaps unattainable.

mondate

Further digging led me to mondate² Its documentation notes

A mondate represents a date as a numeric equalling the number of months since the beginning of the current millennium (the “mondate epoch”). Somewhat arbitrarily, and at the risk of reopening a decade-old debate, “the beginning of the current millennium” is defined as the instant between December 31, 1999 and January 1, 2000. The need for a “mondate” class arises in the area of actuarial analysis, and other areas of financial modeling that need to reconcile to a company’s book of accounts. Its motivation is based on the following presumptions: 1. Business accounting-wise, the closing of the books for a month, quarter, and year are the important milestones for measuring time. 2. For accountants – and actuaries – it is usually not important to measure events on an hourly basis. 3. All events that occur during a business day, up to and including the closing of the books for a day, are all “accounted for” as having occurred “at the same time.” To appreciate the difficulty in measuring the passage of time in days, note that there are typically three fewer days in the first half of the year (January 1 through June 30) than there are in the second half. Yet accountants will say that on June 30th the year is half over. For another example, note that – with the exception of July/August and December/January – the same days of the month for two consecutive months are not one “month” apart if measured in days because, with those exceptions, consecutive months contain differing numbers of days, so which of the two months do you choose as the yardstick? Since changes in accounts over the course of a month, quarter and year are the amounts by which financial results are measured, it is important to be able to measure the passage of time where a year is comprised of twelve months of equal “accounting weight.”

A mondate is simply a real number whose fractional part represents the fraction of the month as of the end of the day. E.g., the fractional part of January 1st = 1/31; the fractional part of February 1st = 1/28 or 1/29, depending on the year. A mondate which is a whole number (i.e., no fractional part) corresponds to a month that is fully completed, whose subsequent month has not yet begun; i.e., the instant in time between one month and the next.

It took a while to digest that, but it seemed promising. The key difference with other approaches is that mondate measures in months from the beginning of its epoch, rather than seconds. This may be relevant in light of the pending GAAP change in 2019 on revenue recognition.

I don’t care what day you recognized that income in April, I just want to be sure that it was April and not May

Here’s a toy example. One important aspect it ignores is Inf values arising from division by zero that came up in my motivating example. Another thing to note is that the package has objects that have to be converted to numeric objects and, in some cases, rounded up. Those are relatively minor and easy to implement.

>     library(mondate)
>     library(lubridate)
>     start  = "2018-02-01"
>     end = "2018-09-01"
>     fully_inclusive <- function (start, end) {
+         start <- mondate(start)
+         end <- mondate(end)
+         open_interval = (end - start) + 1
+         return(as.numeric(round(open_interval,0)))
+     }
> fully_inclusive(start,end)
[1] 8

So, now we’ve looked at something that started on February 1 and finished on September 1 and correctly concluded that eight months were involved.

Garrett Grolemund, Hadley Wickham (2011). Dates and Times Made Easy with lubridate. Journal of Statistical Software, 40(3), 1-25. URL http://www.jstatsoft.org/v40/i03/↩︎
Dan Murphy (2013). mondate: Keep track of dates in terms of months. R package version 0.10.01.02. https://CRAN.R-project.org/package=mondate ↩︎

Is this the original revised data or the revised revised data

Thu, 18 Oct 2018 00:00:00 +0000

Keeping track of the provenance of data can be a challenge, especially when drawing on published sources. Keeping a record of the origin, the date accessed, the transformations applied (e.g., converting from .xls to cvs and converting character strings such as “$1,250,321.21” to floats or date strings to date objects), subsequent changes, who handled the data object and where it can be found in a repository are all things that enhance the analyst’s own ability to reproduce results.

Unfortunately, notes go missing, files get mis-filed and all the other hazards that can befall research can happen. Often, one wishes for R objects with built-in metadata for that purpose.

Using mostattributes() to do this (now preferred)

Update 2015-08-01: Scott Chamerlain at ropensci.org brought **attr* to my attention, which is the built-in way I was looking for originally. He also pointed me to EML, a much more elaborate approach suited for publication projects.

A minimal example

Create data frame and a separate metadata list

fips <- read.csv("https://tuva.s3-us-west-2.amazonaws.com/state_fips_postal.csv", header = FALSE)
fips

##                      V1 V2 V3
## 1               Alabama  1 AL
## 2                Alaska  2 AK
## 3               Arizona  4 AZ
## 4              Arkansas  5 AR
## 5            California  6 CA
## 6              Colorado  8 CO
## 7           Connecticut  9 CT
## 8              Delaware 10 DE
## 9  District of Columbia 11 DC
## 10              Florida 12 FL
## 11              Georgia 13 GA
## 12               Hawaii 15 HI
## 13                Idaho 16 ID
## 14             Illinois 17 IL
## 15              Indiana 18 IN
## 16                 Iowa 19 IA
## 17               Kansas 20 KS
## 18             Kentucky 21 KY
## 19            Louisiana 22 LA
## 20                Maine 23 ME
## 21             Maryland 24 MD
## 22        Massachusetts 25 MA
## 23             Michigan 26 MI
## 24            Minnesota 27 MN
## 25          Mississippi 28 MS
## 26             Missouri 29 MO
## 27              Montana 30 MT
## 28             Nebraska 31 NE
## 29               Nevada 32 NV
## 30        New Hampshire 33 NH
## 31           New Jersey 34 NJ
## 32           New Mexico 35 NM
## 33             New York 36 NY
## 34       North Carolina 37 NC
## 35         North Dakota 38 ND
## 36                 Ohio 39 OH
## 37             Oklahoma 40 OK
## 38               Oregon 41 OR
## 39         Pennsylvania 42 PA
## 40         Rhode Island 44 RI
## 41       South Carolina 45 SC
## 42         South Dakota 46 SD
## 43            Tennessee 47 TN
## 44                Texas 48 TX
## 45                 Utah 49 UT
## 46              Vermont 50 VT
## 47             Virginia 51 VA
## 48           Washington 53 WA
## 49        West Virginia 54 WV
## 50            Wisconsin 55 WI
## 51              Wyoming 56 WY

names(fips) = c("state", "fip", 'id')
require(rjson) # easier to use JSON to write metadata

## Loading required package: rjson

meta <- fromJSON(file = "data/meta.json")
meta

## $Accessed
## [1] "2015-08-01"
## 
## $Analyst
## [1] "Richard Careaga"
## 
## $Contact
## [1] "technocrat@twitter"
## 
## $Preprocessing
## [1] "2015 U.S. police caused deaths through July, reported by The Guardian"
## 
## $Source
## [1] "http://www.theguardian.com/thecounted"

Associate the metadata with the data frame using attr

# invisibly with attr
x <- fips
attr(x, "meta") <- meta

By default, metadata is not displayed

##                   state fip id
## 1               Alabama   1 AL
## 2                Alaska   2 AK
## 3               Arizona   4 AZ
## 4              Arkansas   5 AR
## 5            California   6 CA
## 6              Colorado   8 CO
## 7           Connecticut   9 CT
## 8              Delaware  10 DE
## 9  District of Columbia  11 DC
## 10              Florida  12 FL
## 11              Georgia  13 GA
## 12               Hawaii  15 HI
## 13                Idaho  16 ID
## 14             Illinois  17 IL
## 15              Indiana  18 IN
## 16                 Iowa  19 IA
## 17               Kansas  20 KS
## 18             Kentucky  21 KY
## 19            Louisiana  22 LA
## 20                Maine  23 ME
## 21             Maryland  24 MD
## 22        Massachusetts  25 MA
## 23             Michigan  26 MI
## 24            Minnesota  27 MN
## 25          Mississippi  28 MS
## 26             Missouri  29 MO
## 27              Montana  30 MT
## 28             Nebraska  31 NE
## 29               Nevada  32 NV
## 30        New Hampshire  33 NH
## 31           New Jersey  34 NJ
## 32           New Mexico  35 NM
## 33             New York  36 NY
## 34       North Carolina  37 NC
## 35         North Dakota  38 ND
## 36                 Ohio  39 OH
## 37             Oklahoma  40 OK
## 38               Oregon  41 OR
## 39         Pennsylvania  42 PA
## 40         Rhode Island  44 RI
## 41       South Carolina  45 SC
## 42         South Dakota  46 SD
## 43            Tennessee  47 TN
## 44                Texas  48 TX
## 45                 Utah  49 UT
## 46              Vermont  50 VT
## 47             Virginia  51 VA
## 48           Washington  53 WA
## 49        West Virginia  54 WV
## 50            Wisconsin  55 WI
## 51              Wyoming  56 WY

Metadata has to be invoked by name

attr(x, "meta")

## $Accessed
## [1] "2015-08-01"
## 
## $Analyst
## [1] "Richard Careaga"
## 
## $Contact
## [1] "technocrat@twitter"
## 
## $Preprocessing
## [1] "2015 U.S. police caused deaths through July, reported by The Guardian"
## 
## $Source
## [1] "http://www.theguardian.com/thecounted"

Associate the metadata with the data frame using mostattributes

x <- fips
mostattributes(x) <- list(meta = meta)

The metadata now displays with the data frame

## [[1]]
##  [1] "Alabama"              "Alaska"               "Arizona"             
##  [4] "Arkansas"             "California"           "Colorado"            
##  [7] "Connecticut"          "Delaware"             "District of Columbia"
## [10] "Florida"              "Georgia"              "Hawaii"              
## [13] "Idaho"                "Illinois"             "Indiana"             
## [16] "Iowa"                 "Kansas"               "Kentucky"            
## [19] "Louisiana"            "Maine"                "Maryland"            
## [22] "Massachusetts"        "Michigan"             "Minnesota"           
## [25] "Mississippi"          "Missouri"             "Montana"             
## [28] "Nebraska"             "Nevada"               "New Hampshire"       
## [31] "New Jersey"           "New Mexico"           "New York"            
## [34] "North Carolina"       "North Dakota"         "Ohio"                
## [37] "Oklahoma"             "Oregon"               "Pennsylvania"        
## [40] "Rhode Island"         "South Carolina"       "South Dakota"        
## [43] "Tennessee"            "Texas"                "Utah"                
## [46] "Vermont"              "Virginia"             "Washington"          
## [49] "West Virginia"        "Wisconsin"            "Wyoming"             
## 
## [[2]]
##  [1]  1  2  4  5  6  8  9 10 11 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28
## [26] 29 30 31 32 33 34 35 36 37 38 39 40 41 42 44 45 46 47 48 49 50 51 53 54 55
## [51] 56
## 
## [[3]]
##  [1] "AL" "AK" "AZ" "AR" "CA" "CO" "CT" "DE" "DC" "FL" "GA" "HI" "ID" "IL" "IN"
## [16] "IA" "KS" "KY" "LA" "ME" "MD" "MA" "MI" "MN" "MS" "MO" "MT" "NE" "NV" "NH"
## [31] "NJ" "NM" "NY" "NC" "ND" "OH" "OK" "OR" "PA" "RI" "SC" "SD" "TN" "TX" "UT"
## [46] "VT" "VA" "WA" "WV" "WI" "WY"
## 
## attr(,"meta")
## attr(,"meta")$Accessed
## [1] "2015-08-01"
## 
## attr(,"meta")$Analyst
## [1] "Richard Careaga"
## 
## attr(,"meta")$Contact
## [1] "technocrat@twitter"
## 
## attr(,"meta")$Preprocessing
## [1] "2015 U.S. police caused deaths through July, reported by The Guardian"
## 
## attr(,"meta")$Source
## [1] "http://www.theguardian.com/thecounted"

My now deprecated approach

Metadata are surprisingly easy to create through S4 classes, which was my original thought.

The Rise of the Objects

After encountering S4¹ classes in the sp² and related packages, I hoped to find an R package that provided a similar way of attaching metadata to data frames as the geospatial tools used to attach features to a coordinates data frame. The search was unsuccessful, so the next step was to look into S4 to write something on my own. Several sources helped: Gatto, Genolini and the omnipresent Wickham.

A Simple Example Use

In using U.S. state map data, polygons are sometimes keyed to state names, sometimes to postal codes and sometimes to a census FIPS code. A data frame with all three and the merge function allows easily adding a different key to state identity to data to match the merge field to be used with the related geospatial file.

fips <- read.csv("https://tuva.s3-us-west-2.amazonaws.com/state_fips_postal.csv", header = FALSE)

fips

##                      V1 V2 V3
## 1               Alabama  1 AL
## 2                Alaska  2 AK
## 3               Arizona  4 AZ
## 4              Arkansas  5 AR
## 5            California  6 CA
## 6              Colorado  8 CO
## 7           Connecticut  9 CT
## 8              Delaware 10 DE
## 9  District of Columbia 11 DC
## 10              Florida 12 FL
## 11              Georgia 13 GA
## 12               Hawaii 15 HI
## 13                Idaho 16 ID
## 14             Illinois 17 IL
## 15              Indiana 18 IN
## 16                 Iowa 19 IA
## 17               Kansas 20 KS
## 18             Kentucky 21 KY
## 19            Louisiana 22 LA
## 20                Maine 23 ME
## 21             Maryland 24 MD
## 22        Massachusetts 25 MA
## 23             Michigan 26 MI
## 24            Minnesota 27 MN
## 25          Mississippi 28 MS
## 26             Missouri 29 MO
## 27              Montana 30 MT
## 28             Nebraska 31 NE
## 29               Nevada 32 NV
## 30        New Hampshire 33 NH
## 31           New Jersey 34 NJ
## 32           New Mexico 35 NM
## 33             New York 36 NY
## 34       North Carolina 37 NC
## 35         North Dakota 38 ND
## 36                 Ohio 39 OH
## 37             Oklahoma 40 OK
## 38               Oregon 41 OR
## 39         Pennsylvania 42 PA
## 40         Rhode Island 44 RI
## 41       South Carolina 45 SC
## 42         South Dakota 46 SD
## 43            Tennessee 47 TN
## 44                Texas 48 TX
## 45                 Utah 49 UT
## 46              Vermont 50 VT
## 47             Virginia 51 VA
## 48           Washington 53 WA
## 49        West Virginia 54 WV
## 50            Wisconsin 55 WI
## 51              Wyoming 56 WY

names(fips) = c("state", "fip", "id")

Documenting something so simple seems like overkill until the inevitable question arises is that even right? I used a simple JSON file to make³ a record

require(rjson)
meta <- fromJSON(file = "data/meta.json")
meta

## $Accessed
## [1] "2015-08-01"
## 
## $Analyst
## [1] "Richard Careaga"
## 
## $Contact
## [1] "technocrat@twitter"
## 
## $Preprocessing
## [1] "2015 U.S. police caused deaths through July, reported by The Guardian"
## 
## $Source
## [1] "http://www.theguardian.com/thecounted"

The functions rquired for creating an S4 class are in {base}, so there are no more dependencies.

# create the class
Mframe <- setClass("Mframe",slots = c(meta = "list", data = "data.frame"))
# instantiate it
mf <- Mframe(data = fips, meta = meta)
# show it
str(mf)

## Formal class 'Mframe' [package ".GlobalEnv"] with 2 slots
##   ..@ meta:List of 5
##   .. ..$ Accessed     : chr "2015-08-01"
##   .. ..$ Analyst      : chr "Richard Careaga"
##   .. ..$ Contact      : chr "technocrat@twitter"
##   .. ..$ Preprocessing: chr "2015 U.S. police caused deaths through July, reported by The Guardian"
##   .. ..$ Source       : chr "http://www.theguardian.com/thecounted"
##   ..@ data:'data.frame': 51 obs. of  3 variables:
##   .. ..$ state: chr [1:51] "Alabama" "Alaska" "Arizona" "Arkansas" ...
##   .. ..$ fip  : int [1:51] 1 2 4 5 6 8 9 10 11 12 ...
##   .. ..$ id   : chr [1:51] "AL" "AK" "AZ" "AR" ...

Elaborating this approach to an entire dataset of related objects would be more challenging but this simple application imposes minimal extra work beyond slightly different functions to access the data payload.

mf@data

##                   state fip id
## 1               Alabama   1 AL
## 2                Alaska   2 AK
## 3               Arizona   4 AZ
## 4              Arkansas   5 AR
## 5            California   6 CA
## 6              Colorado   8 CO
## 7           Connecticut   9 CT
## 8              Delaware  10 DE
## 9  District of Columbia  11 DC
## 10              Florida  12 FL
## 11              Georgia  13 GA
## 12               Hawaii  15 HI
## 13                Idaho  16 ID
## 14             Illinois  17 IL
## 15              Indiana  18 IN
## 16                 Iowa  19 IA
## 17               Kansas  20 KS
## 18             Kentucky  21 KY
## 19            Louisiana  22 LA
## 20                Maine  23 ME
## 21             Maryland  24 MD
## 22        Massachusetts  25 MA
## 23             Michigan  26 MI
## 24            Minnesota  27 MN
## 25          Mississippi  28 MS
## 26             Missouri  29 MO
## 27              Montana  30 MT
## 28             Nebraska  31 NE
## 29               Nevada  32 NV
## 30        New Hampshire  33 NH
## 31           New Jersey  34 NJ
## 32           New Mexico  35 NM
## 33             New York  36 NY
## 34       North Carolina  37 NC
## 35         North Dakota  38 ND
## 36                 Ohio  39 OH
## 37             Oklahoma  40 OK
## 38               Oregon  41 OR
## 39         Pennsylvania  42 PA
## 40         Rhode Island  44 RI
## 41       South Carolina  45 SC
## 42         South Dakota  46 SD
## 43            Tennessee  47 TN
## 44                Texas  48 TX
## 45                 Utah  49 UT
## 46              Vermont  50 VT
## 47             Virginia  51 VA
## 48           Washington  53 WA
## 49        West Virginia  54 WV
## 50            Wisconsin  55 WI
## 51              Wyoming  56 WY

#Note @ delimiter; mf$state would also work, but hasn't been set up in the class definition
sessionInfo()

## R version 4.0.0 (2020-04-24)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Pop!_OS 20.04 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rjson_0.2.20
## 
## loaded via a namespace (and not attached):
##  [1] compiler_4.0.0  magrittr_1.5    bookdown_0.20   tools_4.0.0    
##  [5] htmltools_0.5.0 yaml_2.2.1      stringi_1.4.6   rmarkdown_2.3  
##  [9] blogdown_0.20   knitr_1.29      stringr_1.4.0   digest_0.6.25  
## [13] xfun_0.16       rlang_0.4.7     evaluate_0.14

See S3-style Objects and S4-class Objects in the R documentation with help(S4)↩︎
Pebesma, E.J., R.S. Bivand, 2005. Classes and methods for spatial data in R. R News 5 (2), http://cran.r-project.org/doc/Rnews/. Roger S. Bivand, Edzer Pebesma, Virgilio Gomez-Rubio, 2013. Applied spatial data analysis with R, Second edition. Springer, NY. asdar ↩︎
Alex Couture-Beil (2014). rjson: JSON for R. R package version 0.2.15. rjson ↩︎

Data Science in the Board Room and C Suite

Wed, 17 Oct 2018 00:00:00 +0000

Arthur C. Clarke, best remembered for his screenwriting role in 2001: A Space Odyssey is also the source of a famous quotation

Any sufficiently advanced technology is indistinguishable from magic.

We can think of data science as an advanced technology that draws heavily on statistics that have developed in the past decade to exploit the huge gains in power and reduction in cost of computing. Like any rapidly developing technology, approaches, methods, results and usefulness are all over the map. It would be hard to find an executive who has not heard or read glowing tributes to many of the successful applications of data science.

Accounts of failure and disappointment are harder to come by, if for no other reason that they don’t make good press and a shortage of spectacular damage to the unfortunate enterprises that couldn’t find the secret sauce.

It’s been 20 years since the last time we have been enthralled by vast new horizons in technology that appeared in the dotcom boom. Some of that potential has materialized into vast empires; most of the effort disappeared, stories we didn’t hear about until the dotbust.

Relatively few in executive management are data scientists, nor do they need to be. But they should know enough to understand the pitfalls.

Everyone understands cherry picking (aka selection bias). The executive needs to keep that in mind when reading the latest success story.

Every public company has an audit committee of the board of directors, a CFO and a CEO with some combination of finance education, experience and acumen. They worry rightly, about the data stream measured in dollars and cents. How collected? How recorded? How reconciled? How tested? How accurate?

Data governance has been an evolving corporate tool over the past several years, recognizing the hazards of not knowing what you actually do know. I’ll round out Donald Rumsfield with the unknown knowns. It can get ugly when a court decides that a corporate defendant had constructive knowledge of a crucial adverse fact – because there it was right in its own business records, but buried.

Data science, of course, involves data. It may be corporate data, commissioned data or public data. Like all the rest of the data the danger exists that if you know it, you own it. Where did it come from? When? How was it accessed? Who accessed it? How was it transformed? Was it sampled? Were the samples retained? What version control was applied? What tests were performed? Aren’t those test statistics data, too?

Money is fungible, data is not. Does the difference argue for less stringent or more stringent internal controls on the application of data science? I don’t have the answer, and I’m not privy to any internal debates. I do know of at least one substantial organization that is struggling to harmonize the Sarbanes-Oxley processes of a recent acquisition with those of the new parent.

We may see Chief Data Science Officers or even audit committee members with data science literacy. The Industrial Age had well developed protocols for R&D. Data science hasn’t yet matured to the point where it’s susceptible to that level of discipline. At some point, it will need to be.

Cognitive Bias and the Data Scientist

Thu, 11 Oct 2018 00:00:00 +0000

As data scientists, we are not immune from cognitive bias. We do a lot to minimize it, we may even try to quantify it. But it is a part of our protoplasm and, especially, the protoplasm of our untrained clients.

A good place to start is the Monty Hall Problem. The pre-historic Let’s Make a Deal game show on broadcast television featured a host, Monty Hall, and contestants who were offered the chance to win fabulous prizes if they picked the right curtain, 1, 2, or 3. No clues were provided, it was pure guesswork. The possibilities were either a fabulous all expense paid Caribbean cruise, sometimes a consolation microwave and there was always a joke prize, such as a goat.

When the contestant made her choice, Monty did not open the curtain, but made an offer: Give up the chance to see what was behind your curtain? Or choose between the two curtains? And then he opened one curtain to show the goat or the microwave. Keep your curtain or switch?

This put the contestant in an emotional race state between greed and risk aversion. How bad would I feel if I gave up the fabulous prize for either the joke or the consolation prize. As any data scientist would expect, one-third of the time the contestant suffered regret by discovering that she had selected either the goat or the microwave and lost the chance at the dream of a lifetime.

After a while, the popular media asked a woman, reputed to be a super-genius, for the solution, to which she replied that statistically it made sense to take the offer to give up your initial choice and switch. No! The consensus was that there was a 50-50 chance that the contestant already had the winner, which had to be behind one of the two curtains. The concensus, sadly, included many statisticians who should have known better.

Parts of the statistical community, in common with the comman man, raised the irrefutable objection that as between the two curtains there was a 50-50 chance that the prize was hidden behind one of them. Which, of course, was a valid assertation as far as it went, which was not far enough.

Ab initio as those fond of showing obsolete erudition might say, or, more appropriately, at the beginning, the contestant had odds of 1:3 to choose the right door. Meaning, she also had odds of 2:3 of choosing the wrong door. She chose sides, as she was compelled to do, as to which was the right door. She knew that the joint probability of the other two doors containing the prize were 2:3.

Now, comes the hard part. She already knew that only one of the other doors could have the real prize if she didn’t, and since there were two of them, the chances were 50-50, between them. Gaining the knowledge of which one of the doors did not have the winning 50% chance changed her estimate of the odds of the remaining door relative to the open door, but it did nothing (or should have done nothing) to change the estimate of her door relative to the joint probability of the other two doors.

More simply, she started with a 1:3 chance of winning. She still has a 1:3 chance of winning, but she now knows that the remaining alternative has a 2:3 chance of winning. Therefore, she doubles her chances by switching.

Do the math. It’s not hard, except emotionally. A lot of data science problems have that potential.

In the words of Paul Simon

A man sees what he wants to see and disregards the rest.

Ordinary Least Squares Linear Regression Walkthrough for Beginners

Sun, 30 Sep 2018 00:00:00 +0000

Ordinary Least Squares Linear Regression

Ordinary linear least squares regression is one of the first statistical methods that anyone learns who is trying to understand how two sets of numbers relate. It has an undeserved reputation of being able to foretell the future, but it is mathematically tractable, simple to apply, and often yields either directly usable results or signals the need for more sophisticated tools.

The idea is simple enough, find the line that passes through points in Cartesian (x,y 2-dimensional space) that minimizes the total distance (another over-simplification) between each point and the line. In slightly informal terms

\[y = mx + b\]

where y is the dependent variable, x is the independent variable, m is the correlation coefficient, or slope and b is the intercept, the value at which the slope intersects the y axis.

This neglects the term \[\epsilon\] arising from measurement error, random variation or both. I’ll set that aside for this discussion.

A regression, such as miles per gallon against horsepower, may give you plot of the paired point values (x,y) something like the following, indicating a trend.

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	39.94	0.7175	55.66	1.22e-187
horsepower	-0.1578	0.006446	-24.49	7.032e-81

Fitting linear model: mpg ~ horsepower
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
392	4.906	0.6059	0.6049

Even better, the model summary indicates a high coefficient of correlation, R², with a p-value (the unfortunately named “significance”) that is vanishingly small, and so unlikely due to chance.

In another post, I’ve discussed interpreting the model summary, but I haven’t mentioned the graphical methods that are a critical part of understanding what the data are telling you.

Ideally, in the two plots on the left hand side, we would like to see points clustered along the centerline. On the upper right, we would like to see most of the points – the residuals obtained from the distance of a point pair from the dashed line that represents a normal distirubtion – line up. On the lower right, we don’t want to see a curve being “pulled up” by far off outliers.¹

The departures from the ideal are easy enough to spot. What to do about them? Let’s look at the initial data again, this time without the trend line.

How would you draw a line by hand through the points? Would it be two straight lines? A decay curve?

Sometimes, we can improve fit through transforming the scale of one or both variables. A commonly used transformation is to express one variable as a natural logarithm (which is the default in R; other bases are available, such as decimal – log10()). For this dataset a log-log transformation has a slight advantage.

This results in an improvement of R² and a large decrease in the standard error. The plots show minimal non-linearity, the residuals are now very close to being normally distributed, and outliers show negligible effect.

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	6.961	0.1215	57.3	5.168e-192
log(horsepower)	-0.8418	0.02641	-31.88	1.128e-110

Fitting linear model: log(mpg) ~ log(horsepower)
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
392	0.1793	0.7227	0.722

Sometimes it’s not even this subtle

Here is the famous, contrived “anscombe” dataset that shows why plotting should be the first step in any linear regression. There are four datasets and four models, with practically identical output.

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	3	1.125	2.667	0.02573
x1	0.5001	0.1179	4.241	0.00217

Fitting linear model: y1 ~ x1
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
11	1.237	0.6665	0.6295

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	3.001	1.125	2.667	0.02576
x2	0.5	0.118	4.239	0.002179

Fitting linear model: y2 ~ x2
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
11	1.237	0.6662	0.6292

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	3.002	1.124	2.67	0.02562
x3	0.4997	0.1179	4.239	0.002176

Fitting linear model: y3 ~ x3
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
11	1.236	0.6663	0.6292

	Estimate	Std. Error	t value	Pr(>\|t\|)
(Intercept)	3.002	1.124	2.671	0.02559
x4	0.4999	0.1178	4.243	0.002165

Fitting linear model: y4 ~ x4
Observations	Residual Std. Error	$R^2$	Adjusted $R^2$
11	1.236	0.6667	0.6297

Looking before you leap:

Credits

Software (R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.)

data Auto dataset from Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani (2017). ISLR: Data for an Introduction to Statistical Learning with Applications in R. R package version 1.2. https://CRAN.R-project.org/package=ISLR; contrived data from Anscombe, Francis J. (1973). Graphs in statistical analysis. The American Statistician, 27, 17–21. doi: 10.2307/2682899, an included dataset in R.

trendline plot H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016.

formating of linear regression summary Gergely Daróczi and Roman Tsegelskyi (2018). pander: An R ‘Pandoc’ Writer. R packageversion 0.6.2. https://CRAN.R-project.org/package=pander

The technical explanation of the plots I’ve save for another post.↩︎

Deboning linear regression output in Excel

Tue, 18 Sep 2018 00:00:00 +0000

A while back (https://goo.gl/1W11Zu), I outlined interpretation of the output of a multiple linear regression of data on Seattle area housing prices (https://www.kaggle.com/harlfoxem/housesalesprediction?login=true), which provides a convenient way to illustrate the usual output of a multiple linear regression model output in R. This is a 21K dataset with 19 variables on housing characteristics and sales price. It’s a cruddy model, used solely to pick apart the different data presented. Today, it’s Excel’s turn.

Disclosure: I’m not a fan of GUI for most applications. I find it slower and more error prone. Doing this replication elicited the usual grumbles, along with annoyance that multiple independent variables had to be in adjoining columns. Not hard to do, but … .

For comparison, here is the output from the R example, with the addition of an analysis of variance (ANOVA) table that Excel provides by default, but R doesn’t.

library(readr)
library(knitr)
library(kableExtra)
print("Data Structure")

## [1] "Data Structure"

house <- read_csv('https://tuva.s3-us-west-2.amazonaws.com/kc_house_data.csv',comment = '')

## Parsed with column specification:
## cols(
##   .default = col_double(),
##   id = col_character(),
##   date = col_datetime(format = "")
## )

## See spec(...) for full column specifications.

spec(house)

## cols(
##   id = col_character(),
##   date = col_datetime(format = ""),
##   price = col_double(),
##   bedrooms = col_double(),
##   bathrooms = col_double(),
##   sqft_living = col_double(),
##   sqft_lot = col_double(),
##   floors = col_double(),
##   waterfront = col_double(),
##   view = col_double(),
##   condition = col_double(),
##   grade = col_double(),
##   sqft_above = col_double(),
##   sqft_basement = col_double(),
##   yr_built = col_double(),
##   yr_renovated = col_double(),
##   zipcode = col_double(),
##   lat = col_double(),
##   long = col_double(),
##   sqft_living15 = col_double(),
##   sqft_lot15 = col_double()
## )

mod <- lm(price ~ bedrooms + bathrooms + sqft_living + sqft_lot + yr_built, data = house)
summary(mod)

## 
## Call:
## lm(formula = price ~ bedrooms + bathrooms + sqft_living + sqft_lot + 
##     yr_built, data = house)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1784413  -131451   -16905   100357  3933748 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  6.006e+06  1.298e+05  46.281   <2e-16 ***
## bedrooms    -7.096e+04  2.246e+03 -31.592   <2e-16 ***
## bathrooms    8.114e+04  3.730e+03  21.755   <2e-16 ***
## sqft_living  3.044e+02  2.998e+00 101.544   <2e-16 ***
## sqft_lot    -3.388e-01  4.119e-02  -8.225   <2e-16 ***
## yr_built    -3.057e+03  6.686e+01 -45.730   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 245800 on 21607 degrees of freedom
## Multiple R-squared:  0.552,  Adjusted R-squared:  0.5519 
## F-statistic:  5325 on 5 and 21607 DF,  p-value: < 2.2e-16

anova(mod)

## Analysis of Variance Table
## 
## Response: price
##                Df     Sum Sq    Mean Sq   F value    Pr(>F)    
## bedrooms        1 2.7696e+14 2.7696e+14  4585.896 < 2.2e-16 ***
## bathrooms       1 5.3190e+14 5.3190e+14  8807.207 < 2.2e-16 ***
## sqft_living     1 6.6776e+14 6.6776e+14 11056.774 < 2.2e-16 ***
## sqft_lot        1 5.0788e+12 5.0788e+12    84.094 < 2.2e-16 ***
## yr_built        1 1.2630e+14 1.2630e+14  2091.268 < 2.2e-16 ***
## Residuals   21607 1.3049e+15 6.0394e+10                        
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Here’s the corresponding output from Excel 2016 running under Parallels on an Apple laptop.

Excel Output

Let’s take the differences line-by-line

Summary Output: There may be an option tail to add it, but Excel doesn’t give you the formula for the model, the dependent and independent variables involved.
Excel analysis of variance (ANOVA), separately available in R is provided.
Regression Statistics:

Excel adds “multiple R,” which is the square root of R Squared. It’s a measure of the goodness of predicting prices from the model, which can be calculated in R, but is not normally given in the output of the table. I couldn’t find any way in Excel of specifying whether to use Kendall’s tau or Spearman’s rho statistic.
Excel’s remaining reports overlap those of R, except for the omission of detail of residuals and the F statistics.
Excel adds confidence intervals, which are available in R separately. It’s not clear what the difference between the pairs of column represents, and the 95% confidence interval is normally expessed in terms of a value for 92.5% and 97.5%.
Most of the results differ from those given my R for the same data. Except in the cases of the 0 valued p-levels for the t statistic, these are not large, and it is beyond the scope of my post here to figure out why. But apparently different computations are involved.

Cognitive styles of data science

Mon, 10 Sep 2018 00:00:00 +0000

Yihui Xie of RMarkdown fame has a long, thoughtful post (https://goo.gl/WjA7t8) on a debate for and against working in various types of interactive notebooks, such as Juypter for the Python platform, IDEs like RStudio in its so-called “notebook mode” and the old school heavy code commenting, among other approaches. Although that’s the primary subject, he weaves in some astute observations of the differences among “data scientists” based on whether their perspective grows from data origins or is aimed at ultimately producing software.

He associates R with a data orientation and Python with a software engineering orientation:

I feel a major difference between the R culture and Python culture is that Python users seem to create code more often, whereas R users often use code. There seems to be a strong atmosphere of software engineering in the Python world: in the beginning was the custom class (with methods). For R users, in the beginning was the data.

This puts a finger on something that has been lurking in my own thinking. Data science caught on as a catchy phrase to describe the ways we have invented to deal in useful ways with what millenials grew up on – the vast increase of interconnectedness possibilities, digital signal encoding all kinds of phenomena in petabyte increments, accompanied by varying signal-to-noise ratios. The term data science today is on a par with natural philosophy in the Enlightenment, before the study of the world around us settled into discrete and specialized aspects (think of genetic neuroscience, perhaps). The aim of data science is, of course, how we extract meaning from all of these petabyte rivers flowing around us.

Data science is inescapably interdisciplinary. No one of us can wield all the tools necessary. There must be the seekers after truth on the ground, to anchor results in the real world. They are seldom the same as the mathematicians who influence the statisticians who set the computer science types to design algorithms and assess their running times, who may also, but not always, be the computer engineer working to make that feasible by distributed processing or the software engineer writing efficient, fault-tolerant code to implement it. The software engineer may turn to a designer for creating the most human-effective ways of getting data in and taking meaning out. Analysts may be constantly asking what we could be missing or whether it’s possible that an organization has selected a metric for control that is being gamed. It’s no wonder that we can’t agree on the “best” tools or that there is an explosion of infrastructure software as service.

What attracted me to science as an undergraduate was curiosity. I was drawn to programming in my early career by long frustration with manual calculation (it wasn’t before I was 40 that I discoverd that I’m mildly numerically dyslexic, prone to transposition errors). In programming there was, at the beginner’s level, the comfort of certainty. A garage startup during the dotcom boom (no V.C. ever knocked) gave the wonderful experience of what was then a high-speed connection (a fractional T-1 line), and the trial-and-error of getting my domain name records right, servers properly configured and operating and becoming modestly well known for troubleshooting Netscape http/https, mail, news, proxy, certificate and LDAP software.

After my initial exposure to ER data science during the 2007-2008 financial panic, I read Tukey’s classic Exploratory Data Analysis. It was something of a second childhood – this stuff is fun! That’s an aspect that will always attract me, poking under rocks to see what crawls out. As I spent more time in projects aiming to improve electronic discovery in litigation, analysis came to the fore. A copy of email file folders of 148 Enron executives converted from Lotus Notes and Microsoft Outlook into 500,000 plain text files. The Old Law tradition was to have drones read (or at least scan) every line, looking for smoking guns. That’s dumb. A good half were duplicates, half of the rest consisted of usual outside email gunk. Continuing to narrow, I found only about 35,000 emails between senders and receivers who engaged in mutual correspondence (more detail at https://goo.gl/f1jyMK) and introduced me to graph (network) analysis and measures of closeness, centrality, identification of cliques.

All of which finally comes to the point that the right tool depends on what you are doing at the moment. It can be as simple as a plain text log of what you saw, where you saw it, what you tried and ideas or as (relatively) formal as a package. Unlike Middle Earth, we lack one ring to rule them all.

WaPo has some effective visualizations

Sun, 09 Sep 2018 00:00:00 +0000

One of the annoyances of online newspapers such as the NYT and WaPo is that they recycle content, usually off-the-news-cycle. But this sometimes gets offset by finding something you missed the first time.

In a recent article on six common occcupations that have changed from above-average to below average pay, the 2018-09-04 article by Andrew Van Dam and Heather Long is illustrated by some effective techniques.

One technique is using the colors in the caption to serve as a legend to color coding in the graphic.

The other is the use of two views of data separated by a caption giving the common variable identifications.

A lot of design thought goes into communicating with economy, and these are two good examples.

Deboning simple linear regression output in TensorFlow

Tue, 04 Sep 2018 00:00:00 +0000

A simple linear regression example in TensorFlow(TM) is given at (https://www.tensorflow.org/tutorials/keras/basic_regression). By default, it produces none of the outputs of linear regression, other than MAE or MSE, on request through a print statement.

Testing set Mean Abs Error: $2788.86

Time spent searching the documentation turned up no references to “intercept”, “coefficient”, “F-statistic”, “t-value,” or “p-value.”

I am only beginning to dip my toes into TensorFlow(TM). I suspect that it exists not so much as a suite of modeling tools so much as a platform to express them in n-dimensional space very efficiently. Like the output of the example itself, some users seem concerned only with the spread between the model prediction and results when testing (https://web.stanford.edu/class/cs20si/2017/lectures/notes_03.pdf).

There must be a way to use stats packages in Python to specify a model, pass it to TF, have it processed, and retrieve the full diagnostics. Otherwise, it seems too much like trust me, would I … .

Go Not Considered Harmful

Sun, 02 Sep 2018 00:00:00 +0000

I had my first exposure to programming in a FORTRAN class with real punchcards, back in the day, before I knew that GOTO was considered harmful. The exposure was brief and left me wondering whether Russian verbs of motion might not be easier to study, after all.

In the years to follow, I ran the occasional canned deck with my datacards through overnight batch queues, but did not return to anything that could be called programming until I acquired an HP-25 calculator with a macro language, followed shortly by an HP-97 with little magnetic strips on which to record them. And then along came Jobs (and Steve) with a shiny new toy that was tempting but an impossible burden on the family budget.

A few years later, our fortunes improved and I was the proud owner of an Osborne 1 and began to look at Beginner’s All-Purpose Symbolic Instruction Code. Since I had missed Professor Dijestra’s admonition, I happily proceeded to produce spaghetti code. A replacement PC gave me the opportunity to try my hand at Pascal, which imposed greater order, but somewhat more than I was really ready to keep.

Then I heard about Unix by way of a computer show vendor booth of Cromenco, showing their Cromix implementation. I was intrigued, but becoming wary of the future of the much touted S-100 bus of the day. A couple of years later, Digital Equipment Corporation (DEC) released a desktop version of the famed PDP-11 and a Unix version 7 derivative was available to replace the native OS.

After devouring bash and the standard utilities, I took up K&R 1, and began a long-lasting noviate into C. When it came to pointers, though, my cranium began to experience memory leaks. I never did any heavy lifting in C, but it proved a lifetime pass to imperative/procedural programming concepts.

After a long hiatus, I found perl easy to pick up, and vi never leaves your fingers. After another hiatus, Ruby and Python were equally welcoming. Ancient use of dBase II (the purloined property of Jet Propulsion Laboratory) made MySQL intelligible.

R was something of a challenge because it’s mainly a collection of functions. Thanks to the tidyverse, I am now able to more or less fluently combine them. But the functional orientation of R didn’t really kick in until I looked at Haskell, sentential logic disguised as a programming language. I expect that Vulcan children are taught it from pre-school.

Over the past 10 years, I’ve worked with enough moderately large (up to 500K records with 50 variables) running interpreted language routines that took over 24 hours to complete on an old 8-core Mac Pro with 12 GB, to know that the day would come when I’d need more than a passing acquaintance with a compiled language.

Which one?

My first impulse was to revisit C, “a pleasant, expressive and versatile language for a wide variety of programs … [t]hat wears well as one’s experience with it grows.” (Brian Kernighan and Dennis Ritchie in their intro to K&R I). More of the world runs on C than you could read the source code in n lifetimes. Modern C is a subset of C++, so it’s feasible to write runtime links to the compiled code and fold them into R packages.

On the other hand, there’s C++, the Calvinist offspring of C. From what I’ve found from many hours of study, its purpose is to keep programmers from making mistakes. That’s a dealbreaker for me. There’s a big difference between writing relatively small programs designed to make your life easier in other work and being a programmer writing operating systems and complex platforms. So, for example, perfectly good code in C++14 won’t compile in C++11, because the right way to program has been revealed from on high by a later working group.

Then I thought, why not Go? Impeccable pedigree. (The guy at Google who said “complexity is multiplicative” was a prime mover in the project.) Less baggage. Looks, in fact, like an updated C. Go has not been greeted with universal acclaim. See, e.g., Why Go’s design is a disservice to intelligent programmers (http://nomad.uk.net/articles/why-gos-design-is-a-disservice-to-intelligent-programmers.html), which, among other sins, finds it too verbose.

As a not-a-programmer, the harsh assessment doesn’t bother me. So far in my reading of The Go Programming Language by Donovan and Kernighan (yes, that one), I’ve found a combination of the familiarity of C, a design principle of radical simplicity, and useful exercises that I actually take the time to complete and debug.

More of a hope and a prayer than a rational reason to pursue Go as my go-to compiler is TensorFlow, which has a Go API that is said to provide a convenient tool to developing in Python then getting compilation for when you’re read to deploy.

I suppose that a foolish consistency would cause me to reject this path because TensorFlow, itself, isn’t written in Go. But, in any event, the purpose of all languages is to communicate. We’ll see how well I make myself understood in Go.

The Unbearable Lightness of Coloring Maps

Fri, 31 Aug 2018 00:00:00 +0000

The Federal Bureau of Investigation (FBI) has lead responsibilities for protecting the United States against domestic crimes initiated abroad. These responsibilities include protection against “covert actions by foreign governments to influence U.S. political sentiment or public discourse” (https://www.fbi.gov/investigate/counterintelligence/foreign-influence). It has illustrated its mission description with a world map that appears to classify nations by the threats they pose.

The map has no description or legend, but uses several shades of blue to distinguish among countries.

A reasonable interpretation is that the color coding goes from light, representing the least threat, to dark blue, representing the greatest threat, if indeed this is a threat assessment. After all, how big a threat could lightest blue colored Greenland possibly pose?

Canada is somewhat darker, but is exactly the shade of, surprise, China. (You can easily measure websafe color values). That seems somewhat strange. But not as strange as the identical darker blue colors shared by Russia (no surprise) and the United States. It’s possible, I suppose, that these two countries share the same attributes on something. Molybidium reserves? But definitionally, how can the U.S. pose a foreign influence effect on itself?

Besides, although few people suffer from blue-light color blindness, it can be difficult to make fine distinctions among shades of blue.

Here is a filtered version of the original that addresses but does not represent a good approach to the problem, for which I’d need the underlying data.

In this color scheme, the good guys are green, the mild threats are a very low intensity purple, and the bad actors are in increasingly dark shades of purple. Ugly, I know, but easy to explain and interpret. The oceanic colors are a by-product of unnecessary decorative flourishes in the original.

A data scientist who simply says “here are the numbers on a colored map” is not doing her job. Color communicates, and it had better do a good job of conveying an accurate understanding of the data it represents. Fortunately, there is a rich suite of tools, such as ColorBrewer (http://colorbrewer2.org/#type=sequential&scheme=BuGn&n=3) to help the vast majority of data scientists who aren’t trained as graphic designers.

AI and the Oracle at Delphi

Thu, 30 Aug 2018 00:00:00 +0000

I attended a nice Meetup last night, and one of the speakers quoted Turing’s description of a program able to modify its own instructions as a good description of artificial intelligence (AI) and another had a nifty typology relating all the standard tools in data science to their role in AI. This set me to thinking about the standards to which AI should be held.

I began a thought experiment:

Here, AI, is a dataset. I want to find out X. What do you suggest is the best way?

and AI would respond with one of three answers:

This is a simple problem. Here are the standard solutions. You can read about the advantages and disadvantages through the help menu or you can select one to run.
This is a complicated problem, composed of several simple problems. The following combinations have proven helpful. You can read about the advantages and disadvantages through the help menu or you can select one to run.
This is a complex problem. Either more data is required or you will need further research in the domain for solution approaches. Standard tools can be expected to yield different results with the same input.

It doesn’t appear we’ve reached my heart’s desire yet. In a paper that I wished I had saved, I read a criticism of neural networks that they can operate as black boxes, producing perfectly useful results, but providing no traceable method as to how.

That’s not fair! After all, sauce for the goose …

Science is what we understand well enough to explain to a computer. Art is everything else we do.

Donald Knuth, in the forward to A=B (https://www.math.upenn.edu/~wilf/AeqB.pdf).

Why shouldn’t we hold AI to the same standard, and require it to explain to humans?

Finally, I am reminded of Croesus, a king in Asia Minor, not he of the Golden Touch. There were geopolitical issues with the Persian Emperor Cyrus, and Croesus made war plans. Like any good ruler of the time, he consulted the Oracles. A common report is

If you go to war with the Persians, you will a destroy a great empire.

Spoiler: Croesus lost big.

Herodotus’ Histories has a more likely version of the Delphic oracle

Wait till the time shall come when a mule is monarch of Media;
Then, thou delicate Lydian, away to the pebbles of Hermus;
Haste, oh! haste thee away, nor blush to behave like a coward.

(https://sourcebooks.fordham.edu/ancient/herodotus-creususandsolon.asp)

The king managed to torture this result into a favorable auspice.

As successful as AI is proving at pattern recognition and classification, for example, we should take care not to rely heavily on results given without explanation.

How to debone linear regression output in Python

Wed, 29 Aug 2018 00:00:00 +0000

This post makes a trio with two most recent blogs on the same subject for R and Excel, this time in Python usking the sklearn package.

The example given in the official documentation (https://goo.gl/GGmUKK) is somewhat sketchy and uses a different dataset than the previous two examples. Although the example dataset is different, my main purpose is to illustrate the default output.

                        OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.518
Model:                            OLS   Adj. R-squared:                  0.507
Method:                 Least Squares   F-statistic:                     46.27
Date:                Wed, 29 Aug 2018   Prob (F-statistic):           3.83e-62
Time:                        08:39:07   Log-Likelihood:                -2386.0
No. Observations:                 442   AIC:                             4794.
Df Residuals:                     431   BIC:                             4839.
Df Model:                          10
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const        152.1335      2.576     59.061      0.000     147.071     157.196
x1           -10.0122     59.749     -0.168      0.867    -127.448     107.424
x2          -239.8191     61.222     -3.917      0.000    -360.151    -119.488
x3           519.8398     66.534      7.813      0.000     389.069     650.610
x4           324.3904     65.422      4.958      0.000     195.805     452.976
x5          -792.1842    416.684     -1.901      0.058   -1611.169      26.801
x6           476.7458    339.035      1.406      0.160    -189.621    1143.113
x7           101.0446    212.533      0.475      0.635    -316.685     518.774
x8           177.0642    161.476      1.097      0.273    -140.313     494.442
x9           751.2793    171.902      4.370      0.000     413.409    1089.150
x10           67.6254     65.984      1.025      0.306     -62.065     197.316
==============================================================================
Omnibus:                        1.506   Durbin-Watson:                   2.029
Prob(Omnibus):                  0.471   Jarque-Bera (JB):                1.404
Skew:                           0.017   Prob(JB):                        0.496
Kurtosis:                       2.726   Cond. No.                         227.
==============================================================================

Source: User JARH on Stackoveflow (https://goo.gl/vZ3UWP), using the same dataset as the official documentation.

The output is divided into three sections.

OLS REGRESSION RESULTS lacks the “call,” identifying the dependent and independent variables. It contains less information on residuals than R and adds some additional information. The date/time stamp is a gift during the exploratory data phase before the analyst has started more formal procedures.

Four additional substantive fields are added

Covariance Type is a useful reminder whether the the requirements for a reliable linear regression are satisfied totally or not (“nonrobust”).
Log-likelihood specifies which measure was applied to deal with the role of error terms
AIC – the Akalike information criterion – is a “score” to assess the quality of competing models
BIC – the Baysian inforation criterion – performs a similar role

The second section, below the “===============” line is similar to R and Excel in providing coefficients and standard error. Like Excel, it provides a confidence interval (but in standard 92.5/97.5 form for the 95% confidence interval) that has to be run separately in R. Unlike the other two programs it omits the convenient asterisk coding of significance levels.
The third section, below the “================” has information omitted from both R and Excel or in a different form.

The Omnibus test is the F statistic from analysis of variance (ANOVA), included in Excel and separatly available in R. Skew and kurtosis describe asymmetries in the distribution curve. Durbin-Watson tests for autocorrelation. It is separately available in R; I haven’t checked in Excel. Jarque-Bera tests whether the data is normally distributed. It is separately available in R; I haven’t checked in Excel. The condition number is a measure of collinearity among the independent variables, which can have the effect of overemmphasizing redundant variables. It is separately available in R; I haven’t checked in Excel.

As sometimes said of economists, you can lay them end-to-end, and they would still point in different directions. Sklearn’s community of designers make different choices from those of R and the proprietary team of Microsoft Excel. Which is better or worse is a matter of personal taste, so long as each provides some functionality to produce the same output separately from the default for linear regression output.

If you don’t regularly use all three, I hope these guides have been useful.

How to debone the output of a linear regression model

Mon, 27 Aug 2018 00:00:00 +0000

I was recently puzzling over an academic paper in the social sciences with a multiple linear regression model that seemed off. (Communication with the authors educated me on considerations that resolved my concerns.)

During the course of my puzzling, I compared what I was seeing as model output in the paper and what I expected to see based on experience. The embarrassing revelation dawned that I didn’t actually know what all those components meant.

So, I decided to find out. Data on Seattle area housing prices provide a convenient way to illustrate the usual output of a multiple linear regression model output. This is a 21K dataset with 19 variables on housing characteristics and sales price.

## [1] "Data Structure"

## Parsed with column specification:
## cols(
##   .default = col_double(),
##   id = col_character(),
##   date = col_datetime(format = "")
## )

## See spec(...) for full column specifications.

## cols(
##   id = col_character(),
##   date = col_datetime(format = ""),
##   price = col_double(),
##   bedrooms = col_double(),
##   bathrooms = col_double(),
##   sqft_living = col_double(),
##   sqft_lot = col_double(),
##   floors = col_double(),
##   waterfront = col_double(),
##   view = col_double(),
##   condition = col_double(),
##   grade = col_double(),
##   sqft_above = col_double(),
##   sqft_basement = col_double(),
##   yr_built = col_double(),
##   yr_renovated = col_double(),
##   zipcode = col_double(),
##   lat = col_double(),
##   long = col_double(),
##   sqft_living15 = col_double(),
##   sqft_lot15 = col_double()
## )

## 
## Call:
## lm(formula = price ~ bedrooms + bathrooms + sqft_living + sqft_lot + 
##     yr_built, data = house)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1784413  -131451   -16905   100357  3933748 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  6.006e+06  1.298e+05  46.281   <2e-16 ***
## bedrooms    -7.096e+04  2.246e+03 -31.592   <2e-16 ***
## bathrooms    8.114e+04  3.730e+03  21.755   <2e-16 ***
## sqft_living  3.044e+02  2.998e+00 101.544   <2e-16 ***
## sqft_lot    -3.388e-01  4.119e-02  -8.225   <2e-16 ***
## yr_built    -3.057e+03  6.686e+01 -45.730   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 245800 on 21607 degrees of freedom
## Multiple R-squared:  0.552,  Adjusted R-squared:  0.5519 
## F-statistic:  5325 on 5 and 21607 DF,  p-value: < 2.2e-16

Well now! I pick a handful of variables off the top of my head and presto! My model explains over 50% of the variation in price. Am I good or what?

Not really, and I’ll explain all the bonehead errors in another post (didn’t normalize, didn’t split into training and test sets, used variables that are not truly independent, didn’t check the distribution of residual errors and a host of other sins). Let’s talk about the information you get from a multiple linear regression results table.

The Call is an explicit statement of the model.
The Residuals show a summary distribution of the differences between the observations and predictions. In this example half of the residuals are about, low, up to $17,000 and and half are above, including one whopper that’s almost $4 million high.
The Coefficients include, in addition to the coefficient and standard error, a t value and the probability that the coefficient is greater that the absolute value of the t value. It is essentially equivalent to the significance level – the probability, which is extremely low in this example, of the results being due to random effects.
The Signif. codes are for convenient reference to the size of the significance level.
The Residual standard error indicates the average amount that the observed values deviate from the trend line, in this case approximately $246,000.
Multiple R Squared, indicates the proportion of variance explained by the coefficient, holding the other coefficients constant.
Adjusted R Squared, accounts for possible interaction among the other independent variables.
The F-statistic addresses the question of whether there is, in fact, no relation between one or more of the independent variables and the coefficient. An F-statistic that approaches 1 precludes us from rejecting the null hypothesis of no relation and to conclude that there is, in fact, no relationship at all between one or more of the independent variable with the dependent. The degrees of freedom equals the number of data points, minus 1. The p-value, or significance, gives the probability that the F-statistic is merely the result of random chance. The extremely low p-value in this example indicates that the F-statistic is highly unlikely to be a chance artifact.

As I said before, there are defects in my model that render these results garbage, and my table should be accompanied by a discussion that the conditions required to make multiple linear regression are actually unsatisfied. It’s solely to illustrate the information provided in the table.

Just goes to show you don’t really understand something you haven’t tried to explain.

The plural of data science is not formulae

Tue, 14 Aug 2018 00:00:00 +0000

The WSJ report of academic research on career hot streaks has snagged a lot of LinkedIn eyeballs today (Career ‘hot steaks’ are real 1h ago • 3,526 readers).

Reading the paper (https://goo.gl/2Mwqjn), you can see that it is an observational study of a convenience sample (these were the people about whom data were available, in the thousands or tens of thousands, an impressive collection).

Two mysteries: Why were the data not stratified into a training set and a test set? How could temporal auto-correlation fail to be addressed?

\[f(x)=y\]

isn’t data science. And observational data of temporal patterns in somewhat simplistic metrics of an arbitrarily selected population tells you menos de zero about individuals outside the population.

Finally, you need a theory to account for your results – a chain of reasoning that takes you from what you see to what you can reliably predict with your shiny new analysis.

The academics have no robes, and I am failing adequately to defer to their superior credentials and posts. Too bad, because I would liked to have learned something useful.

Reading further, there’s no indication that the nominal prices were inflation adjusted. Whatever the latest auction price was recorded was used.

Cinema directors include assistant directors and art directors, who have primarily administrative, not creative, roles. The authors measure director success by the IMDB user ratings. I don’t know how IMDB handles it, but it’s easy to imagine that as users come and go, and tastes change, the scores from that source may not share a common scale.

Academics get measured by citation. Often for good reason, a researcher will cite her previous related publications. Are those “self-citations” included or excluded?

We aren’t even told whether the three measures of “goodness” were standardized to have a mean of 1 and standard deviation of 0.

The data and code, however, are provided at https://goo.gl/p8WYqH. Because the code won’t load, due to a couple of type errors, the results can’t be checks. Although scipy.stats is imported, it’s not used, so it’s unlikely that the data were standardized.

Such is Nature, I suppose.

Analytic Data Scientists -- know when to fold 'em

Sun, 12 Aug 2018 00:00:00 +0000

In the days of our forebearers, if a tool broke, you were likely to fix it yourself. Long time passing, of course, the shadetree mechanic has been an endangered species. And no one would hire a carpenter who didn’t own his own tools.

I’ve been one to use my own tools. I have to admit to being, at times, cavalier about corporate policies strictly forbidding the use of employee owned computers, at least in the building. (Why it was ok remotely without VPN is something of a mystery.)

I took Admiral Hopper’s adage to heart

It’s easier to ask forgiveness than than it is to get permission.

Even with the overhead of being my own sysadm, I saved at least 90 minutes a week in boot time. The corporate XP system took (I kid you not) 20 minutes to boot and present a working GUI. In that time my Mac Pro could boot, connect to slow off-site WiFi and give me time to triage my morning email through the corporate webmail system.

However, as tools proliferate, the balance between overhead and maintenance has changed.

TIL: The default Mac and Ubuntu compilers are C++11. Not C++14 nor C++17.

In a project to get literate in C++ I learned that the preferred way to declare a constant was

 const int age {25};

Since everything is up-to-date in Seattle, that’s what I did, saddening the compiler and receiving a multitude of error messages. I hit upon the idea of good ’ol C style

const int age = 25;

but I wondered what other slithery bugs awaited me.

Through the kindness of strangers, I learned of a compiler flag. I know, I know, RTFM.

And then it struck me. I’m not a programmer. I never wanted to be a programmer. I just wanted to be able to read and understand a program in a language as important as C++ and maybe write some simple programs that gave a hemibel boost to execution.

As an analytic data scientist, I want to spend my time thinking about data and ways to extract useful information from them. I can’t do that if I’m spending hours trying to figure out why syntactically valid software won’t compile.

I’ll be cultivating friendships with data engineers and programmers.

Nathan Yau on the challenges of thematic maps

Thu, 09 Aug 2018 00:00:00 +0000

If you are a paying member of Flowing Data (well worth the modest subscription), you will have received his email, reproduced at a paywalled portion of the site.

He gives a detailed critique of a precinct-by-precinct map of the 2016 presidential election, nationwide, showing the number of votes for Trump and Clinton and the nearest precinct in which the opposing candidate was in the majority.

He compares what the map purports to show with what it actually does show, which is the raw total of votes (ho-hum, we’ve known that for a long time) and the percentages. What the map fails to show is population density.

More concretely, a 100-person precinct that voted 20 percent for Trump still had more Trump votes than a 10-person precinct that voted 100 percent for Trump.

In other words, the larger precinct is “blue,” while the smaller precinct is “red.” But twice as red by headcount.

My own observation: Combined with our innate overperception of area and failing to weight voting results by percentage, we perceive a sea of red surrounding a few islands of blue. However, the national total blue vote was higher.

Newtonian Data Sciences

Mon, 06 Aug 2018 00:00:00 +0000

Advanced machine learning and AI strike me as deterministic finite state machines with an error term. Properly tuned, the same input should produce much the same output every time. Like Newtonian physics, they can be complicated, but deconstructible into simple components. Like Newtonian physics, as well, they are highly accurate and reliable within their domain.

What happens when we start applying the same tools to complex systems? What are the complex systems of importance that refuse to yield identical results with identical inputs? Health in its totality (not just diagnosing specific disorders), political stability, climate change are a few to come to mind. Our tools may appear to work¹ but then fail spectacularly. Now would be a good time to start worrying about this class of problems.

Value at risk (VAR) was a quantitative method to estimate risk exposure to owning financial assets. It was supposed to be a conservative check on the portfolio composition. It failed spectacularly in 2008 when implicit and explicit assumptions about the financial environment changed, and the interconnectedness of financial institutions around the world had been overlooked in favor of a unitary view of The Market.↩︎

An error message is as much of a question as it is an answer

Sun, 05 Aug 2018 00:00:00 +0000

A question on datascience.stackexchange asked the reason for a Python syntax error

SyntaxError: can't assign to literal

Another poster had the correct change to fix the error but the explanation didn’t reflect what the OP needed to understand the problem behind the problem, which was failing to be namespace aware – he was trying to perform an operation on an undefined object.

When you find the answer to a question you don’t understand, you can expect to see it again.

Row collapsing

Sun, 20 May 2018 00:00:00 +0000

Untidy happens

It’s not always possible to store everything in a tidy but hugmongous data store. So, we have things like SQL foreign keys.

Unfortunately, the database may not have been set up with future tidy in mind. Or, database access is only provided through CSV files.

Here’s an example of what can happen. We begin with some basic information, as tidy as we could wish.

> patients
# A tibble: 3,064 x 6
     INC_KEY   AGE GENDER MALE  FEMALE ADULT
       <int> <int> <chr>  <lgl> <lgl>  <lgl>
 1 150238916    59 Male   TRUE  FALSE  TRUE 
 2 150193078    37 Male   TRUE  FALSE  TRUE 
 3 150580164    29 Female FALSE TRUE   TRUE 
 4 150917895    82 Female FALSE TRUE   TRUE 
 5 150376887    58 Female FALSE TRUE   TRUE 
 6 150508153    35 Male   TRUE  FALSE  TRUE 
 7 150415059    21 Male   TRUE  FALSE  TRUE 
 8 150184182    24 Female FALSE TRUE   TRUE 
 9 150318438    20 Male   TRUE  FALSE  TRUE 
10 150508148    64 Male   TRUE  FALSE  TRUE 
# … with 3,054 more rows

A separate file contains variables of interest

> d_codes
# A tibble: 25,643 x 2
     INC_KEY DCODE 
       <int> <chr> 
 1 150284827 922.2 
 2 150284827 867.0 
 3 150284827 835.00
 4 150284827 808.51
 5 150284827 808.43
 6 150284827 808.2 
 7 150284827 805.6 
 8 150004383 924.8 
 9 150004383 919.0 
10 150004383 901.0 
# … with 25,633 more rows

Now, of course we can left_join the two, but, of course, we get

  str(patients)
  Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   28042 obs. of  23 variables:
   $ INC_KEY     : int  150580164 150580164 150580164 150580164 150580164 150580164 150580164 150580164 150580164 150580164 ...
   $ YEAR        : num  2015 2015 2015 2015 2015 ...
   $ AGE         : int  29 29 29 29 29 29 29 29 29 29 ...
   $ ADULT       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
   $ INTERVENTION: logi  FALSE FALSE TRUE FALSE FALSE FALSE ...
   $ SURGERY     : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
   $ EXPIRED     : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.0       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.1       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.2       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
   $ 808.3       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.4       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.41      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.42      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.43      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.49      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.5       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.51      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.52      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.53      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.59      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.8       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
   $ 808.9       : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...

Notice the duplicated INC_KEY field in the first line of str()

This is the dreaded duplicate row problem.

Many approaches (nested tibbles, for example), failed either through limitations or my own misunderstanding on how they work. I did, however,find a solution based on the the fortunate fact that the variables to be combined were all logical.

The 800 series in the table above represent types of pelvic fractures. Because the source data records for each INC_KEY, or patient identification, for these and other diagnostic codes, there are many more rows of data than there are patients. After a simple extraction of the 800 codes, the de-duplication proved surprisingly simple.

# deduplicate
dedup_fractures <- fractures %>% group_by(INC_KEY) %>% summarize_all(mean)
fractures <- dedup_fractures %>% group_by(INC_KEY) %>% mutate_if(is.double, as.logical)

For a TRUE entry, the mean will be 1; a FALSE entry will be 0. The second line simply converts these results back to logical type.

This approach is not strictly limited to logical vectors. For example, you have have a second table with numeric values, such as age, that do not vary. The mean of age 29 is 29 whether you have one or a hundred rows.

About

Thu, 05 May 2016 00:00:00 +0000

Analytic data science with an emphasis on R. I am available for pro bono work to non-profit organizations. However, I am unable to incur travel expense outside of King County, Washington.

Mon, 01 Jan 0001 00:00:00 +0000

About

Analytic data science with an emphasis on R. I am available for pro bono work to non-profit organizations. However, I am unable to incur travel expense outside of King County, Washington.

Mon, 01 Jan 0001 00:00:00 +0000

Composition

$f: X \rightarrow Y$

Just as visualizing tensors degrades as $n > 3$, juggling pieces in an R script can get jiggly when they do not %>% neatly. A post recently reminded me so.

If conceptualized as school algebra ($f(x)=y$) writ large, every discrete R tasks is to find or compose $f$.

# https://community.rstudio.com/t/appending-dataframes-to-list-named-by-variable/77689/5
suppressPackageStartupMessages({library(dplyr)
                                library(purrr)
                                library(tidyr)
                              })
f <- function(x) {
  map(1:length(mk_vs()),sub_df) %>% set_names(.,mk_names(mk_vs(),x))
}

mk_synth_df <- function() {
  set.seed(137)
  ID <- sample(1:100, 50, replace=TRUE)
  N <- sample(1:100, 50, replace = TRUE)
  ratio <- rnorm(50, mean = 0.5, sd=0.1)
  data.frame(ID,N,ratio)
}

mk_list <- function() {
  set.seed(137)
  list("item1" = sample(1:100, 100), "item2" = sample(1:100, 100))
}

mk_vs <- function() {
  set.seed(137)
  v1 <- sample(1:100, 10, replace = TRUE)
  v2 <- sample(1:100, 10, replace = TRUE)
  v3 <- sample(1:100, 10, replace = TRUE)
  list("res1" = v1, "res2" = v2, "res3" = v3)
}

mk_names <- function(x,y) {
  paste(names(x)[1:length(x)],y, sep="")
}

sub_df <- function(x) {
  drop_na(mk_synth_df()[mk_vs()[[x]],])
}

f("_dFList")
#> $res1_dFList
#>    ID  N     ratio
#> 34 47 27 0.4427454
#> 8  96 22 0.5997265
#> 38 71 44 0.6615295
#> 39 14 83 0.4865537
#> 35 15 59 0.4820364
#> 
#> $res2_dFList
#>    ID  N     ratio
#> 30 35 89 0.6159864
#> 48 38 43 0.3970454
#> 13 13 21 0.4866509
#> 43 51 14 0.4953379
#> 14 43 55 0.5479219
#> 22 48 96 0.5725216
#> 4  38 44 0.4549781
#> 6  39 25 0.7056911
#> 
#> $res3_dFList
#>      ID  N     ratio
#> 15   14 89 0.5237554
#> 48   38 43 0.3970454
#> 32   86 89 0.5995506
#> 13   13 21 0.4866509
#> 14   43 55 0.5479219
#> 35   15 59 0.4820364
#> 35.1 15 59 0.4820364

^{Created on 2020-08-27 by the reprex package (v0.3.0)}

Locus: Richard Careaga

Composition

Category theory, first steps

A minor lubridate snip

Look before you count

Even more wrong

Not even wrong

What happened to the streak? rle: run length encoding

HSL in R, Chapter 1

An R Companion to Chapter 1 of Hosmer, Lemeshow and Sturdivant’s Applied Linear Regression

Source data

Code for text tables and figures

Section 1.1

Table 1.1

Figure 1.1

Table 1.2

Table 1.3

Table 1.4

Figure 1.3

Commentary

Equation 1.1

Equation 1.2

Equation 1.3

Equation 1.4

Logistic regression

Why learn about logistic regression?

HSL doesn’t have code examples, but it does have tables, figures and exercises

Also in preparation

Metadata for datasets

Where does this dataset come from?

Using mostattributes() to do attach metadata

A minimal example

Create data frame and a separate metadata list

Associate the metadata with the data frame using mostattributes

Now metadata is displayed by default

Missing Inaction--locating data holes

Some real world data, masked and anonymous

What’s missing?

The not applicables

Why do we care?

What can we do?

How do we remove the NAs?

But wait!

Dependent variables of interest are all zero valued for some observations

How do we identify them?

Even more

Flash cards

Hidden missing data

Review of R Cookbook, 2nd Ed.

eDiscovery is pattern recognition and eyeballs are expensive

The compiler will tell you what the user cannot

Less coding and more analysis

Color Map Atlas for Continuously Scaled Maps

Color, Design and Communication

An example of continuous data

Three Problems: One Major, One Important and One Nuisance

The Major Problem: Counts Need Transformation

The Important Problem

Alpha adjustment

Beyond Blue Palette Solutions

Viridis palette

ggplot’s scale_fill_gradient can also over-ride default colors

RColorBrewer

Palette Snippets

Credits

Commercial data science: a parable of downsides of crisp deployment

R and Haskell, meant for each other?

n?

Month arithmetic in R, a quick guide

Here are questions you might want to ask about months:

First, Excel.

Python

MySQL

R

Base

lubridate

mondate

Is this the original revised data or the revised revised data

Using mostattributes() to do this (now preferred)

A minimal example

ggplot’s `scale_fill_gradient` can also over-ride default colors