| Sample Size | |
|---|---|
| Rejections | |
| Resume | n |
| A | 256 |
| B | 256 |
Resume Thunderdome
A job application experiment
TL;DR
This is a story about my strategy to find another job after getting laid off and what I learned in the process.
The winning version is… the sleek-one page resume 🥳 it had a statistically significant lower rejection rate (at the 0.05 level) and a non-significant higher interview rate
I also re-applied to a few roles with a different email address and resume. Interestingly, at least one of these companies rejected me when I applied with the two-page resume (version A) but contacted after I applied with the sleek, one-page resume (version B).
Oh sh!t
Upon returning from parental leave in October 2024 I was notified that my team (~20 people) had been let go. I’m cool in a panic so I immediately began to formulate a plan.
DDoA attack
Have you heard of a distributed denial of service (DDoS) attack in networking? It’s basically when a system is flooded with requests. My plan was like a DDoS attack.
With the help of an AI amigo I came up with a name for my plan: Distributed Deluge of Applications attack (DDoA). The plan was to apply to as many jobs as possible, analyze the data, fine tune things, and repeat. Basically, an initial brute force approach followed by an optimization procedure to find the best strategy.
The plan had three components:
- Apply to as many jobs as possible across various levels (e.g., data scientist - VP)
- Record every application
- Analyze the data
After two weeks of applying to jobs and about 300 job applications later my wife reformatted my traditional, two-page resume into a sleek one-pager. Can you guess what we did next? If you said, AB test which resume version received more responses than you guessed correctly.
Overview
Let’s start with a base understanding of what’s happened over the course of the experiment.
- Days in the market: 43
- Start date: Sep 26, 2024
- Latest date: Nov 14, 2024
- Total applications: 821
- Total rejections: 237 (30%)
- Total interviews: 28 (3%)
So, in 43 days I’ve applied to 821 jobs. I’ve been explicitly rejected 237 times. And I’ve had 28 conversations with recruiters or hiring managers. Not too shabby. It seems my resume opens doors. I just have to practice ABC.
Thunderdome rules
Contestants or Resumes versions:
- A: Traditional, two-pager
- B: Sleek one-pager
Metrics to be analyzed:
- Rejection rate
- Interview rate
The winner is the resume version with:
- The lower rejection rate
- The higher interview rate
Resources:
- An Introduction to Categorical Data Analysis (Agresti 2018)
- Testing & Experimentation, A Population Proportion Example (Rodriguez 2024)
Rejections
To be fair to the resume versions, the rejections analysis and any rejection metric only considers applications I didn’t interview for. If I was rejected after an interview that has nothing to do with the resume version. That’s on me.
| Overall Results | |||
|---|---|---|---|
| Rejections | |||
| Resume | Not Rejected | Rejected | Rejection Rate |
| A | 149 | 107 | 42% |
| B | 176 | 80 | 31% |
Now for some stats fun. Let’s set the null hypothesis as \(H_0: \pi_B \ge \pi_A\) and the alternative hypothesis to \(H_1: \pi_B \lt \pi_A\). Where \(\pi\) denotes the rate of interest for a particular resume version. In plain English, we’re testing the belief that the one-page version of my resume (B) will have a lower rejection rate than the traditional two-page version (A).
Fisher's Exact Test for Count Data
data: tmp
p-value = 0.008451
alternative hypothesis: true odds ratio is greater than 1
95 percent confidence interval:
1.146231 Inf
sample estimates:
odds ratio
1.578408
2-sample test for equality of proportions without continuity correction
data: tmp
X-squared = 6.1415, df = 1, p-value = 0.006602
alternative hypothesis: greater
95 percent confidence interval:
0.03588716 1.00000000
sample estimates:
prop 1 prop 2
0.4179688 0.3125000 Takeaways:
- The winner is: Resume version B
- Version A rejection rate: 41.8%
- Version B rejection rate: 31.2%
- The result of the hypothesis tests: Both tests reject the null hypothesis (statistically significant difference)
- The results of the statistical analysis suggests that the odds of a rejection using version A are 2 times the odds using version B
Interviews
Interviews are considered as any point of contact. It doesn’t matter if I made it to the final round or it was just a conversation with the recruiter. The goal of the assessment is to determine which resume version opens doors.
| Sample Size | |
|---|---|
| Interviews | |
| Resume | n |
| A | 265 |
| B | 265 |
| Overall Results | |||
|---|---|---|---|
| Interviews | |||
| Resume | Not Interviewed | Interviews | Interview Rate |
| A | 256 | 9 | 3.4% |
| B | 255 | 10 | 3.8% |
The null hypothesis is \(H_0: \pi_B \le \pi_A\) and the alternative hypothesis is \(H_1: \pi_B \gt \pi_A\). Where \(\pi\) denotes the rate of interest for a particular resume version. In plain English, we’re testing the belief that the one-page version of my resume (B) will have a higher interview rate than the traditional two-page version (A).
Fisher's Exact Test for Count Data
data: tmp
p-value = 0.5
alternative hypothesis: true odds ratio is less than 1
95 percent confidence interval:
0.000000 2.151119
sample estimates:
odds ratio
0.8966695
2-sample test for equality of proportions without continuity correction
data: tmp
X-squared = 0.054589, df = 1, p-value = 0.4076
alternative hypothesis: less
95 percent confidence interval:
-1.00000000 0.02279132
sample estimates:
prop 1 prop 2
0.03396226 0.03773585 Takeaways:
- The winner is: Resume version B
- Version A rejection rate: 3%
- Version B rejection rate: 4%
- The result of the hypothesis tests: Both tests fail to reject the null hypothesis
- The results of the statistical analysis suggests that the odds of getting an interview using version A are 10% lower than the odds using version B
Discussion
Resume versions:
- A: traditional, two-pages
- B: sleek, one-page
Regarding sample sizes - that is, the number of job applications per resume version. 265 jobs were applied to with resume version A. However, 556 jobs have been applied to with resume version B. Since I haven’t continued to apply with resume version A, the analysis only considers the first 265 jobs applied to with version B.
In the beginning only version A existed. After a few weeks my wife created version B. So, applications with resume version A are older than jobs I applied to with version B. As a result, jobs applied to with version A have had more time to be rejected. Given that the age of an application likely impacts the likelihood rejection, it might be worthwhile to explore an analysis that weights applications by their age - like logistic regression with application age as the weights.
Rejections:
- Version A
- min date: Sep 26, 2024
- max date: Oct 13, 2024
- application window: 17 days
- mean age of applications: 40 days
- Version B
- min date: Oct 15, 2024
- max date: Oct 25, 2024
- application window: 10 days
- mean age of applications: 25 days
The mean age calculation is based on the current date, Nov 14, 2024. On average, jobs applied to with resume A had an additional 15 days to be rejected.
Regarding randomization, I didn’t randomly assign a resume version per application because in the beginning version B didn’t exist. However, the availability of any given job at a point in time is random enough for these purposes.
Another thing to consider are the types of roles and the job levels I applied to with each resume. The big difference between resume versions and job types is I applied to fewer data scientist IC roles with version B in favor of ML engineering IC roles. As far as job levels, resume version B slightly favored junior roles than version A (e.g., DS and MLE roles vs senior DS and senior MLE roles). These differences may also account for variations in the rejection rate. The tables below illustrates this point using the rejections data.
| Job Type | ||
|---|---|---|
| Proportion by Resume | ||
| A | B | |
| academia | 0% | 2% |
| ai | 4% | 5% |
| analyst | 7% | 0% |
| analytics | 9% | 13% |
| analytics engineer | 0% | 1% |
| applied scientist | 2% | 1% |
| business intelligence | 0% | 1% |
| data engineering | 0% | 0% |
| data science | 65% | 51% |
| data strategist | 1% | 0% |
| decision science | 1% | 0% |
| marketing science | 1% | 0% |
| ml | 7% | 23% |
| operations research | 0% | 0% |
| other | 0% | 2% |
| product | 1% | 0% |
| research | 1% | 0% |
| statistician | 2% | 0% |
| Job Level | ||
|---|---|---|
| Proportion by Resume | ||
| A | B | |
| 1 | 33% | 42% |
| 2 | 38% | 35% |
| 3 | 9% | 8% |
| 4 | 17% | 11% |
| 5 | 2% | 1% |
| 6 | 1% | 4% |
As noted before, a more comprehensive analysis could account for these differences by including age of the application, job type, and job level.
Conclusions
Overall, the sleek one-page resume (version B) wins because it had a statistically significant lower rejection rate and resulted in the same number of interviews as the traditional, two-page resume (version A).
I’ll refresh the analysis periodically to see how things change.
Appendix
Applications
| Overall Results | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Applications | |||||||||||||||||||
| Level | academia | ai | analyst | analytics | analytics engineer | applied scientist | business intelligence | data engineering | data science | data strategist | decision science | economist | marketing science | ml | operations research | other | product | research | statistician |
| 1 | 0% | 2% | 0% | 6% | 0% | 0% | 0% | 0% | 18% | 0% | 0% | 0% | 0% | 8% | 0% | 0% | 0% | 0% | 0% |
| 2 | 0% | 1% | 0% | 3% | 0% | 0% | 0% | 0% | 24% | 0% | 0% | 0% | 0% | 7% | 0% | 0% | 0% | 0% | 0% |
| 3 | 0% | 1% | 1% | 0% | 0% | 0% | 0% | 0% | 6% | 0% | 0% | 0% | 0% | 2% | 0% | 0% | 0% | 0% | 0% |
| 4 | 0% | 0% | 1% | 0% | 0% | 0% | 0% | 0% | 8% | 0% | 0% | 0% | 0% | 3% | 0% | 0% | 0% | 0% | 0% |
| 5 | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| 6 | 0% | 0% | 0% | 1% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| Overall Results | ||
|---|---|---|
| Applications | ||
| Level | IC | Management |
| 1 | 19% | 18% |
| 2 | 30% | 7% |
| 3 | 10% | 0% |
| 4 | 13% | 0% |
| 5 | 1% | 0% |
| 6 | 0% | 1% |
Rejections
Considers only roles that I did not interview for.
| Overall Results | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rejections | |||||||||||||||||
| Level | academia | ai | analyst | analytics | analytics engineer | applied scientist | business intelligence | data science | data strategist | decision science | economist | marketing science | ml | operations research | other | research | statistician |
| 1 | 0% | 2% | 0% | 6% | 0% | 1% | 0% | 15% | 0% | 0% | 0% | 0% | 7% | 0% | 1% | 0% | 1% |
| 2 | 0% | 2% | 0% | 3% | 0% | 0% | 0% | 21% | 0% | 0% | 0% | 0% | 7% | 1% | 0% | 1% | 0% |
| 3 | 0% | 0% | 2% | 0% | 0% | 0% | 0% | 6% | 0% | 0% | 0% | 0% | 3% | 0% | 0% | 0% | 0% |
| 4 | 0% | 1% | 2% | 0% | 0% | 0% | 0% | 10% | 0% | 0% | 0% | 0% | 3% | 0% | 0% | 0% | 0% |
| 6 | 0% | 1% | 0% | 1% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
| Overall Results | ||
|---|---|---|
| Rejections | ||
| Level | IC | Management |
| 1 | 14% | 21% |
| 2 | 29% | 8% |
| 3 | 9% | 1% |
| 4 | 17% | 0% |
| 6 | 0% | 3% |
Interviews
| Overall Results | ||||
|---|---|---|---|---|
| Interviews | ||||
| Level | analytics | data science | ml | statistician |
| 1 | 7% | 32% | 0% | 4% |
| 2 | 11% | 25% | 7% | 0% |
| 3 | 0% | 11% | 0% | 0% |
| 4 | 0% | 4% | 0% | 0% |
| Overall Results | ||
|---|---|---|
| Interviews | ||
| Level | IC | Management |
| 1 | 25% | 18% |
| 2 | 25% | 18% |
| 3 | 11% | 0% |
| 4 | 4% | 0% |
A quick comparison of interview rates by resume version using all of the data.
Interviewed Not Interviewed
A 9 256
B 19 537
Interviewed Not Interviewed
A 0.034 0.966
B 0.034 0.966Same company and role different resume version
Rejected A
Rejected B 1 0
1 3 0
0 4 9Interestingly, there are some roles that were rejected using resume version A but not B.
date_A date_B
2 2024-09-27 2024-11-10
3 2024-09-28 2024-11-08
7 2024-10-02 2024-10-21
8 2024-10-02 2024-10-21My guess is I’ll eventually be rejected for the version B roles given the content is the same. Unless of course the recruiter likes the sleek, one-page version better 🤞
Interviews A
Interviews B 1 0
1 0 1
0 0 15I noticed that some roles were re-posted after I initially applied. If I originally applied with resume version A, then I re-applied using version B. This seemed to me a good way to test the resume versions. At least one company rejected me when I applied with version A but contacted me for more info when I applied with version B. It’s a small sample size but it’s more evidence towards version B, the sleek one-page resume.
References
Agresti, Alan. 2018. An Introduction to Categorical Data Analysis. 3rd ed. Wiley. https://www.wiley.com/en-us/An+Introduction+to+Categorical+Data+Analysis%2C+3rd+Edition-p-9781119405283.
Rodriguez, Santiago. 2024. “Binomial Example Report.” RPubs. https://rpubs.com/kaladin_stormblessed/binomial_example_report.