Building a serverless “Twitter data stack” for free with dbt and BigQuery (using Python, Google Cloud Functions, and Pub/Sub triggers)

➔ Overview

This project deploys a serverless Python app on Google Cloud Functions and uses a Pub/Sub trigger to pull tweets from the Twitter API, push them to BigQuery, process them with dbt, and then send tweets to people who have gone quieter than usual.

Inspiration

I was inspired recently by a tweet from Claire Carroll:

gonna set up a whole data stack just to:
- extract data from twitter
- load it into a warehouse
- set up a dbt model that detects when the people I follow stop tweeting at their normal frequency
- uses a "reverse ETL" tool to send a twitter DM that says "u ok?"

— Claire Carroll (@clairebcarroll) August 6, 2021

I've been looking for an excuse to dig into some technologies that are new to me, and decided to use this as an opportunity for a weekend project.

The technologies on my short list included Docker and Google Cloud Platform (GCP), especially BigQuery.  Using these services also has the added advantage of making the project completely free (compared to Snowflake, which has no free tier, or AWS, which is well known to have a dangerous free tier that's impossible to understand).

Reflections on USV vs. a16z

Decided to share these thoughts from March 2014:[1]

I was reflecting on Fred Wilson’s comment where he analogized Union Square Ventures (USV) to a boutique clothing store and Andreessen Horowitz (a16z) to Walmart.  That suggestion sets up a jarring contrast but I believe it’s deceptive.  Discussion about the case on a16z brought out some of the reason why the analogy is imperfect, but we never really covered why a16z is a bit of a strange animal.

A better analogy for USV vs. a16z would be a boutique consultancy vs. a big consulting firm—and a few people touched on a16z as a McKinsey analogue.  The dichotomy between boutique and big consulting is fitting because it covers many of the differences we observed between USV and a16z.  In particular, the benefits of scale are quite real in consulting (as are returns per GP at a16z), but a boutique firm may still be a better option for a “power player” who can attract business and charge high rates without the branding of a large firm (Fred Wilson at USV?).

What’s both interesting and puzzling about a16z—and I think what led to so much equivocation in class—is that Marc and Ben are each independently power players.  So why go off and build a scaled monolith where other GPs and an army of professionals drag down the returns you could get with a big-shot boutique?  In that regard, a16z appears truly anomalous because Marc and Ben don’t seem to be doing it for money, fame, power, success, etc.  They seem to believe that scaling the organization allows them to make a bigger impact—an impact they are happy to share, both with their organizational pyramid and through their significant philanthropic efforts.

Thus it seems that a16z breaks the mold because Marc and Ben themselves break the mold.  Having had such incredible success already (and still in their 40s), they are not in it for any of the reasons that many others pursue venture capital.  Instead, they are in it for the game.  [And I’m sure Ben could provide some apropos lyrics.]

Notes

[1] This was written in March after Fred Wilson visited Launching Technology Ventures, a course at Harvard Business School (HBS) taught by Jeff Bussgang.  Originally for an internal HBS blog, I have made only slight edits to clarify for a broader audience.

A/B Testing to Increase Mobile App Profitability

I recently wrote a piece on increasing profitability through A/B testing with Joey Conway.  The public version abstracts a few details but it makes for a great case study on A/B testing because the app we look at (the free version of Joey's Root Checker) has a simple interface with a single call to action, several links to upsell Joey's paid apps, and one unobtrusive ad.  We lay out our methodology, the basic design changes, and the (powerful) impact that we saw as a result of those changes: take a look!

The post was initially for a class at Harvard Business School called Launching Technology Ventures (LTV), taught by Jeff Bussgang of Flybridge Capital.  For those lucky enough to have the opportunity, LTV is an excellent class and Jeff is an incredible instructor.  Tweets related to the class often appear with #HBSLTV.  You can check out our post on the course blog.

Saturation of .ly Domains with English Adverbs

The .ly TLD is uniquely suited for domain hacks because “ly” is a common ending for many English adverbs.  I looked into the saturation of .ly domains suitable for domain hacks by comparing a list of common English words ending in “ly” against the Libyan ccTLD registry.  Then I analyzed the prevalence of these words in Google Books Ngram data to see how the frequency of a word in writing impacts its probably of being registered for a domain hack.

Almost every popular adverb that can make a domain hack on “.ly” has already been registered.  Essentially all common words that remain unregistered fall into one of three undesirable categories: 1) short words that can only be registered by Libyans, 2) words with negative connotations, and 3) words not permitted by the Libyan registrar.