Meetup: Spotify’s Recommendations Lambda Architecture

Meeting announcement from Meetup via the Organizer, Eugene Dvorkin:

At Spotify, one of the ways we surface personalized music recommendations to users is via the Discover page. The recommendations are powered by training large scale Matrix Factorization models on user listening history. Batch hadoop jobs run daily to build latent vectors for users, tracks, artists and albums.

However, this setup doesn’t incorporate intra-day listening history. Furthermore, new users that log into Spotify and stream music will not receive recommendations until their second day using the application. To solve the problem, we leverage Storm and process user listens in real-time; this allows us to surface recommendations to new users as soon as they start listening to songs on Spotify. We shall discuss our framework, how we use Storm and challenges we faced building this system.