| Life stage | Mean | SD | Min | Median | Max | N |
|---|---|---|---|---|---|---|
| Student | 4.9% | 2.0% | 0.0% | 5.0% | 22.6% | 1121 |
| Working Adult | 35.3% | 32.5% | 0.0% | 38.2% | 98.9% | 4135 |
| Elderly | 4.6% | 4.1% | 0.0% | 4.3% | 65.8% | 1838 |
Justice
Wisdom told us what question to ask and what ideal data would look like. Justice asks whether the data we actually have is good enough to answer that question honestly — and if not, exactly how and why it falls short.
What is Justice?
Justice is the second Cardinal Virtue of data science. It governs the gap between the ideal dataset (the Preceptor Table) and the data we actually have. Justice requires us to:
- Check validity. Do our columns actually measure what we intend them to measure? A variable named “income” in one dataset may not mean the same thing as “income” in another.
- Build a Population Table. Expand the Preceptor Table into the full population we care about, adding both the rows we observe (data) and the rows we want to generalize to (Preceptor rows).
- Assess stability. Would the relationship we estimate hold in a different time, place, or economic environment? If the data come from one era and the question refers to another, stability may fail.
- Assess representativeness. Does our sample actually represent the population we care about? Who is systematically absent?
- Check for hidden biases. Are there other factors we haven’t measured that could be driving the results along with income? If so, we can’t be sure the income effect is real.
- Choose a statistical model. Select the right type of equation to describe how income relates to spending — the modeling choice that carries forward into Courage.
Justice does not fix problems — it names them honestly so that Temperance can later explain what we cannot conclude.
One row per household per survey period, drawn from the population of U.S. households across all life stages from roughly 2022 to 2024.
Units: Each row represents one household (or individual, in the student dataset) observed during a single survey period. Households are not tracked over time — each one appears exactly once, like a photograph rather than a video.
Treatment: Annual income — the household’s annual income, which policy can change through tax credits, transfers, or minimum wage increases. Life stage is used to check whether the income effect differs across groups — that comparison is the core question.
Outcome: Discretionary spending share — the percentage of total household spending allocated to leisure and entertainment. This is what we are trying to explain.
Background factors: Life stage and number of children. Life stage is the grouping variable that lets us check whether the income effect differs across groups. Number of children is included because families with more children tend to spend differently, regardless of income level.
Data rows: The 7,094 households from three sources — the Kaggle Student Spending dataset, the Kaggle Customer Personality Analysis dataset, and the 2024 BLS Consumer Expenditure Public Use Microdata — covering students, working-age adults, and elderly households respectively. These are the rows we actually observe.
Quantity of interest: How much does a $10,000 income increase change the share of spending that goes to discretionary goods — and does that change differ across life stages? The key estimates are how steeply each group’s spending line rises (or falls) with income, and whether those slopes are different for students, working adults, and the elderly.
Preceptor rows: All U.S. households we wish to generalize to — students enrolled at any institution nationwide, working adults across all industries and regions, and elderly households including those in institutional care currently excluded from household surveys. These rows exist in the ideal dataset but not in ours.
Visual Population Table
The table below combines the Data rows (what we actually observe) with Preceptor rows (what we wish we could observe). Each household appears in both blocks: once with the spending we actually measured, and once with what spending would have been at income + $10,000. That second column is never directly observed — this is the core challenge of estimating causal effects from real-world data.
| Source | Unit | Life Stage | Current Income | Observed Spending Share | Spending Share at +$10K |
|---|---|---|---|---|---|
| Data | Student A | Student | $15K | 22% | … |
| Data | Working Adult B | Working Adult | $60K | 14% | … |
| Data | Elderly C | Elderly | $30K | 18% | … |
| … | … | … | … | … | … |
| Preceptor | Student A | Student | $15K | 22% | [unobserved] |
| Preceptor | Working Adult B | Working Adult | $60K | [unobserved] | 14% |
| Preceptor | Elderly C | Elderly | $30K | 18% | [unobserved] |
| … | … | … | … | … | … |
In Data rows, the counterfactual is marked … because no one measured it. In Preceptor rows, both cells are shown — but the unobservable one is italicized to mark it as the truth we cannot access. The average causal effect is the average difference between the two spending-share columns, which we can never compute directly because each household contributes only one observed value.
Summary Statistics
The spread (SD) is large relative to the group averages, confirming that households within the same life stage still vary enormously. The averages are useful summaries, but they hide a lot of individual variation.
Sample Composition
| Life stage | Source | N | Age range | Median income |
|---|---|---|---|---|
| Student | BLS_PUMD | 121 | 16–24 | $35,000 |
| Student | Kaggle_BLS | 1000 | 18–25 | $12,252 |
| Working Adult | BLS_PUMD | 1913 | 25–64 | $90,422.60 |
| Working Adult | Kaggle_BLS | 2222 | 28–84 | $51,381.50 |
| Elderly | BLS_PUMD | 988 | 65–88 | $40,954.90 |
| Elderly | Kaggle_BLS | 850 | 65–90 | $65,812.65 |
Who Is In This Data?
Our dataset combines three sources into 7,094 household-level rows:
Who Is Missing?
Justice requires naming the populations excluded from our analysis — not just describing who is included.
Structurally excluded:
- Households with zero reported spending (dropped as likely incomplete interviews)
- Individuals with incomes above $500,000 (trimmed as outliers that distort shares)
- BLS PUMD respondents under 25 who are not students (only 121 rows; likely underrepresented)
Systematically absent:
- Homeless and housing-insecure individuals — not captured by any household survey
- Institutionalized elderly (nursing homes, assisted living) — BLS surveys non-institutional households only
- International students — the Kaggle student dataset skews toward U.S. college enrollment patterns
- Very low-income households who may underreport spending or income in surveys
A generational gap:
The elderly in our data grew up in a fundamentally different economy. Their spending patterns reflect not just their current life stage but Depression-era and post-war savings habits, employer pensions, and healthcare costs from before the Affordable Care Act. A 70-year-old in 2024 is not simply a 30-year-old today projected forward in time — their entire financial history is different.
Does Our Outcome Measure What We Mean?
Validity asks whether pct_discretionary — the share of spending on leisure/entertainment — actually captures “discretionary” behavior as we intend it.
The honest answer is: approximately, but imperfectly.
Across our three datasets, “discretionary” maps to different underlying categories:
| Dataset | What counts as “discretionary” |
|---|---|
| Student (Kaggle) | Entertainment + Personal care + Technology |
| Working adult (Customer Personality) | Wine + Sweets + Gold products |
| Elderly (BLS PUMD) | Entertainment expenditures (ENTERTCQ) |
These are not identical. A student’s “technology” spend includes a laptop for coursework — arguably essential. An elderly person’s “entertainment” may include prescription TV subscriptions that provide genuine social connection. The categories are the best available proxies, but the concept is broader than any single column.
Life stage labeling is also imperfect. Life stage is inferred from age and dataset source rather than verified directly — a 22-year-old not in school could be counted as a student simply because they appear in the student dataset. This labeling error is small but not zero, and could slightly understate the true differences between groups.
Distribution of the Outcome
The distribution is lopsided (more households cluster at lower shares) with heavy overlap between groups. This tells us the model will capture the average pattern but leave a lot of individual variation unexplained — households within the same life stage still spend very differently from one another.
Would This Model Hold Under Different Conditions?
Stability asks whether the relationships we estimate here would replicate in a different time, place, or economic environment.
Several reasons to be cautious:
Cross-year mixing: The student data is from approximately 2023, the working adult data from 2022–2023, and the BLS PUMD from 2024. The 2022–2024 period included elevated inflation, rising interest rates, and post-COVID normalization of spending patterns. These are not “normal” economic conditions, and the model’s estimates reflect this specific environment.
Post-pandemic shifts: Spending on food at home vs. away from home, entertainment, and healthcare shifted substantially during 2020–2023. A model trained on this window may not generalize to pre-2020 patterns or to a future lower-inflation environment.
Who Does This Sample Actually Represent?
Representativeness asks whether the sample generalizes to the population we care about. Here there are three separate representativeness questions — one per dataset.
Students: The Kaggle student dataset has 1,000 records of unknown origin. There is no description of which schools or countries are represented, and no information about how students were selected. It likely overrepresents students who are financially aware enough to track their spending — not a random cross-section of all students.
Working adults: The Customer Personality Analysis dataset is from a marketing campaign, not a representative survey. It overrepresents customers of one company and almost certainly excludes the very poor (who are not customers) and the very wealthy (who are not the firm’s target market).
Elderly: The BLS Consumer Expenditure Survey is a government probability survey — the most rigorously designed of the three datasets. However, it only covers households living independently, so the most frail elderly (in nursing homes or assisted living) are not included.
Are Other Factors Distorting Our Results?
This section asks whether there are other factors — ones we haven’t measured — that could be driving the relationship between income and spending, making it hard to isolate income’s true causal effect.
The honest answer is: partially, but not fully.
Income tends to go hand-in-hand with other household characteristics that also affect spending: - Wealth: A household with the same income but more savings will spend differently — savings affect spending independently of current income. - Spending habits: Naturally frugal households tend to earn more and spend less on discretionary goods. This can make income look more powerful than it really is. - Region: Cost-of-living differences mean the same dollar income buys different amounts in different cities.
These factors mean we cannot fully separate the income effect from the other things that tend to come with higher income. Our estimates show the observed income pattern — pointing in the right direction, but not a perfectly clean experiment. The main question — whether that pattern differs across life stages — still stands even under this limitation.
What type of model should we use?
Standard linear regression.
Why?
Discretionary spending share is a number that can range continuously — it is not a yes/no outcome, a count, or a category. Linear regression is the natural choice here: it estimates how the average spending share changes as income rises, while accounting for life stage and family size. The histogram above shows that spending shares are spread out fairly smoothly within each group, so a straight-line model is a reasonable fit.
The specific form our model uses:
\[\mathbb{E}[Y_i \mid X_i] = \beta_0 + \beta_1 \cdot \text{Income}_i + \beta_2 \cdot \mathbf{1}[\text{Working Adult}_i] + \beta_3 \cdot \mathbf{1}[\text{Elderly}_i]\] \[+ \beta_4 \cdot \text{Children}_i\]
This says: predicted spending share = a starting value + income effect + life-stage adjustments + a children adjustment. Each Greek letter (β) is a number the model estimates from the data.
How Much Does the Model Miss?
Even after accounting for income, life stage, and number of children, the model’s predictions are off by about 22.9 percentage points for a typical household. In other words, two households that look identical on paper can still spend very differently — the model captures the average pattern, not individual behavior.