TMH #9: Experimentation with Matt Gershoff
Published 19 October, 2021
After the “drought” of the last three episodes, you can finally relax as it’s not just me doing a solo episode. And what an amazing guest I managed to persuade to join me!
Matt Gershoff of Conductrics is a good friend and an eloquent speaker. Above all, he’s so incredibly knowledgeable about experimentation and about statistical inference in general.
All the work we do with data is a subset of statistical inference. We collect data to draw conclusions, which we hope to extrapolate to the experiences and intuitions that we haven’t (yet) measured against. Often this is done through means of passive analysis, which is more related to the traditional “analytics” work done in organizations. Data is collected, and it is mined for insight and for feeding into new hypothesis generation.
The other approach is that of experimentation, where a business question is first formulated, then data is collected specifically for the needs of answering this question, and finally the results are analyzed on the basis of whether there’s (sufficient) evidence to suggest that an answer was found to the question or not.
Matt walks us through the ins and outs of techniques such as A/B testing, the frequentist and Bayesian approaches, and how it’s really up to organizations and the individuals within to make the right calls with how they approach data. No tool or service delivers automatic gratification.
Having said that, I would be remiss to not point out how amazing Conductrics is as an experimentation platform. It promotes transparency and open-source approaches over black boxes and magically produced results. It’s very developer-friendly, being designed and developed with powerful APIs in mind first and foremost, although it does provide a very intuitive user interface for running more traditional A/B test setups with.
We didn’t have enough time to discuss the Conductrics approach, which is why I’m comfortable in sharing this little unsolicited (and definitely not paid-for) shout out to what is my favorite experimentation platform out there!
Anyway, listen to the podcast to find out what trams in Helsinki have to do with understanding the p-value in the frequentist approach..