Las Vegas | June 1-4, 2020
Workshop: Advanced A/B Testing
Monday, June 17, 2019 in Las Vegas
Full-Day: 8:30am – 3:30pm
There is nothing better than a head-to-head A/B test to drive marketing decisions. A/B tests lead to simple analytics that almost everyone in the organization can understand and act on. If you’ve run at least a few A/B tests, this workshop will show you how to use more powerful analytics tools like predictive modeling to get more insight out of your tests. After reviewing the basics of A/B test analysis, we will talk about testing strategies you can use when you have a very large or very small sample sizes. We will cover a pile of advanced techniques including heterogeneous treatment effects, uplift modeling, causal forests, blocking, matching, and stratification. If this sound like a bunch of jargon to you right now, that’s okay. I will demystify the jargon and help you understand when and why you might use these techniques in analyzing your A/B tests. We will also cover Bayesian approaches to A/B testing including test & roll and multi-armed bandits.
Is this for me?
If you know what an A/B test is and you are enthusiastic about learning more advanced analytics techniques, this workshop is for you. My goal is to make sure you have an overall understanding of what these techniques do and when you would use them, without getting bogged down with technical details. By the end of the workshop, you will be ready to turbo-charge your A/B testing program. You may need to collaborate a data scientist execute on some of the concepts, but you will know what it is you want that data scientist to do.
Will there be hands-on-examples?
Yes! During the workshop, I will be working examples in R. If you don’t know R, you can still follow the workshop focus on the concepts. You don’t have to know how to write R code to understand what I’m doing and what the results mean. If you do know some R, all the code will be available in advance and you are welcome to follow along during the workshop. (I’m sad that I won’t have time to teach R to novices in the workshop, but you might check out my book R for Marketing Research and Analytics.)
Takeaways:
- How do you get more out of your A/B tests when your sample sizes are really big or way too small?
- Can I get more lift by deploying my marketing to the right customers or stores?
- How do “classical” and Bayesian approaches to A/B testing differ?
Workshop Outline
A/B testing basics
- Analysis (confidence intervals and p-values)
- Planning (sample size formula)
When your sample size is big
- Slice and dice (heterogeneous treatment effects)
- Find the opportunities (uplift modeling)
When your sample size is small
- Reducing the noise (stratification and matching)
When your ultimate goal is to make a decision
- Go Bayesian (test & roll based on a recent paper by Feit & Berman)
- Get dynamic (multi-armed bandits)
When you can’t randomize [time permitting]
- Fake it (propensity matching, causal forests)
Workshop Pricing
See the registration page for full pricing options.
SAVE $300 when purchasing with a conference ticket!
Instructor
Elea McDonnell Feit, Assistant Professor of Marketing, Drexel University
Elea McDonnell Feit is an Assistant Professor of Marketing at Drexel University and a Senior Fellow The Wharton School at the University of Pennsylvania. Her research focuses on leveraging customer data to make better product design and advertising decisions, particularly when data is incomplete, unmatched or aggregated. Much of her career has focused on developing new quantitative methods and bringing them into practice, first working in product design at General Motors, then commercializing new methods at the marketing analytics firm, The Modellers, and most recently as the Executive Director of the Wharton Customer Analytics Initiative, where she built the academic-industry partnership program. She enjoys making analytics and statistics accessible to a broad audience and has recently co-authored a book on R for Marketing Research and Analytics with Chris Chapman and a DataCamp course on Choice Modeling in R. She regularly teaches popular tutorials and workshops for practitioners on digital marketing, marketing experiments, marketing analytics in R, discrete choice modeling and hierarchical Bayes methods as well as undergraduate, MBA and MS Business Analytics classes at Drexel and Wharton. She holds a PhD in Marketing from the University of Michigan, an MS in Industrial Engineering from Lehigh University and a BA in Mathematics from the University of Pennsylvania.