There is nothing better than a head-to-head A/B test to drive marketing decisions. A/B tests lead to simple analytics that almost everyone in the organization can understand and act on. If you’ve run at least a few A/B tests, this workshop will show you how to use more powerful analytics tools like predictive modeling to get more insight out of your tests. After reviewing the basics of A/B test analysis, we will talk about testing strategies you can use when you have a very large or very small sample sizes. We will cover a pile of advanced techniques including heterogeneous treatment effects, uplift modeling, causal forests, blocking, matching, and stratification. If this sound like a bunch of jargon to you right now, that’s okay. I will demystify the jargon and help you understand when and why you might use these techniques in analyzing your A/B tests. We will also cover Bayesian approaches to A/B testing including test & roll and multi-armed bandits.
If you know what an A/B test is and you are enthusiastic about learning more advanced analytics techniques, this workshop is for you. My goal is to make sure you have an overall understanding of what these techniques do and when you would use them, without getting bogged down with technical details. By the end of the workshop, you will be ready to turbo-charge your A/B testing program. You may need to collaborate a data scientist execute on some of the concepts, but you will know what it is you want that data scientist to do.
Yes! During the workshop, I will be working examples in R. If you don’t know R, you can still follow the workshop focus on the concepts. You don’t have to know how to write R code to understand what I’m doing and what the results mean. If you do know some R, all the code will be available in advance and you are welcome to follow along during the workshop. (I’m sad that I won’t have time to teach R to novices in the workshop, but you might check out my book R for Marketing Research and Analytics.)
A/B testing basics
When your sample size is big
When your sample size is small
When your ultimate goal is to make a decision
When you can’t randomize [time permitting]
Elea McDonnell Feit, Assistant Professor of Marketing, Drexel University
Elea McDonnell Feit is an Assistant Professor of Marketing at Drexel University and a Senior Fellow The Wharton School at the University of Pennsylvania. Her research focuses on leveraging customer data to make better product design and advertising decisions, particularly when data is incomplete, unmatched or aggregated. Much of her career has focused on developing new quantitative methods and bringing them into practice, first working in product design at General Motors, then commercializing new methods at the marketing analytics firm, The Modellers, and most recently as the Executive Director of the Wharton Customer Analytics Initiative, where she built the academic-industry partnership program. She enjoys making analytics and statistics accessible to a broad audience and has recently co-authored a book on R for Marketing Research and Analytics with Chris Chapman and a DataCamp course on Choice Modeling in R. She regularly teaches popular tutorials and workshops for practitioners on digital marketing, marketing experiments, marketing analytics in R, discrete choice modeling and hierarchical Bayes methods as well as undergraduate, MBA and MS Business Analytics classes at Drexel and Wharton. She holds a PhD in Marketing from the University of Michigan, an MS in Industrial Engineering from Lehigh University and a BA in Mathematics from the University of Pennsylvania.
What are cookies?
Cookies, among other things, help to make your visit to our website simpler, more pleasant and more logically structured. Cookies are information data which your web browser automatically stores on your computer’s hard disc when you visit our website. Cookies do not do any damage to your computer’s hard disc and they do not transmit any of your personal data to us.
We use the web analysis service Google Analytics to continuously streamline and optimise our sites for users. For this purpose, pseudonymised user profiles are created and small text files are used that are stored on your computer (‘cookies’). The information produced by the cookie about your use of this website, such as
is transmitted to servers of Google Inc. in the USA and stored there. This information is used to evaluate the use of our website, to compile reports on website activities and to provide other services connected with the use of the website and Internet use for purposes of market research and to enable these Internet sites to be made more user-friendly. If necessary, this information is also transmitted to third parties to the extent that this is prescribed by law or if third parties process the data for us. In no event is the IP address ever brought into association with other data relating to the user. IP addresses are anonymised to prevent them from being matched (IP masking).
Users may configure their browsers to prevent cookies from being installed. However, we would point out that in this case it may not be possible to use all of the functions of this website to the full extent
You can find more information about the web analysis service we use on the website of Google Analytics. You can find instructions for preventing your data being processed by the web analysis service at http://tools.google.com/dlpage/gaoptout?hl=en.
Unsere Websites verwendet sogenannte Re-Targeting-Technologien. Wir nutzen diese Technologien, um für Sie das Internetangebot interessanter zu gestalten. Diese Technik ermöglicht es, Internetnutzer, die sich bereits für unsere Dienstleistungen und Produkte interessiert haben, auf den Websites unserer Partner mit Werbung anzusprechen. Wir sind davon überzeugt, dass die Einblendung einer personalisierten, interessenbezogenen Werbung in der Regel für den Internetnutzer interessanter ist als Werbung, die keinen solchen persönlichen Bezug hat. Die Einblendung dieser Werbemittel auf den Seiten unserer Partner erfolgt auf Basis einer Cookie-Technologie und einer Analyse des vorherigen Nutzungsverhaltens. Diese Form der Werbung erfolgt vollkommen anonym. Es werden keine personenbezogenen Daten gespeichert und es werden auch keine Nutzungsprofile mit Ihren personenbezogenen Daten zusammengeführt. Sie können die Verwendung von Cookies durch Google deaktivieren, indem sie die Seite zur Deaktivierung von Google-Werbung aufrufen. Alternativ können Sie die Verwendung von Cookies durch Drittanbieter deaktivieren, indem sie die Deaktivierungsseite der Netzwerkwerbeinitiative aufrufen.