➔ Overview
This project deploys a serverless Python app on Google Cloud Functions and uses a Pub/Sub trigger to pull tweets from the Twitter API, push them to BigQuery, process them with dbt, and then send tweets to people who have gone quieter than usual.
Inspiration
I was inspired recently by a tweet from Claire Carroll:
gonna set up a whole data stack just to:
— Claire Carroll (@clairebcarroll) August 6, 2021
- extract data from twitter
- load it into a warehouse
- set up a dbt model that detects when the people I follow stop tweeting at their normal frequency
- uses a "reverse ETL" tool to send a twitter DM that says "u ok?"
I've been looking for an excuse to dig into some technologies that are new to me, and decided to use this as an opportunity for a weekend project.
The technologies on my short list included Docker and Google Cloud Platform (GCP), especially BigQuery. Using these services also has the added advantage of making the project completely free (compared to Snowflake, which has no free tier, or AWS, which is well known to have a dangerous free tier that's impossible to understand).
