a couple of weeks ago I read an article they talked about how recent IPL cricket auctions were life-changing for several cricket players that list of players included Ben Stokes who's one of my favorite British cricketers and he was quoted in the article saying I was

following it on Twitter I didn't actually see it live I kept on refreshing my notifications I thought people were tweeting and then I realized that Pune had got me why am I sharing this anecdote with you all well for one I can talk about cricket in this country

without having to explain myself and as you can tell I love talking about cricket but more importantly it ties very closely to our jobs to be done framework and we want to keep our users informed about the world the other thing is it also showcases one of the prime

use cases of Twitter where people come to Twitter to follow a live event and to hear the real-world commentary around it and for both of these use cases my team plays a very central role and we we enable these use cases by notifying users about what's happening in their

world in real time the key to highlight here is their world and real-time and during the talk today we will focus on why these two are important my name is Saul Apothic I manage the notifications team at Twitter I'm here today with Gary Lamb

who's the lead engineer in my team today we will be giving a high-level overview of notifications then we will talk about some of the main challenges we face scaling it and then we'll go over how our infrastructure solves them after that we will go over the evolution of

notifications and broadly there are three categories that we will cover today. We will start with triggered notifications and then we will talk about personalized sign-out and recommendations. So that's not before, before we start, let's do a quick show of hands. Raise your hands if you have used

student notifications in the past. It's awesome, but for those who did not raise your hands, I'll have to find you and start sending you notifications. But for your benefit, I will give a quick recap of the recap of the four main channels we use

to notify our users. The first one is notification timeline. This is an in-app experience. You go to your timeline to see a reverse chronological history of all your notifications. It acts as a pivot for the product. Then there's push. It's a heavily used channel. It ties very

closely to the core of Twitter, which is students live, and these notifications are real-time. One of the most common forms of notification is account notification. Through account notification, you can opt in to be notified every time someone tweets, and that's that comprises of a huge volume

for push. And then there is our SMS channel. Some of you might not be aware, but Twitter actually started as a group send SMS service, and so SMS is still relevant for us. Apart from some product cases, we use it for sending login codes and two-factor authorization and whatnot, and

then finally email. It's the oldest messaging channel on the internet, and it's still pretty relevant for us. Now let's switch gears and talk about some of the main challenges that we face, and they have a bearing on how our architecture has evolved over time. The

The first one I want to highlight is notifications. Our bimodal, what I mean there is, if you were to plot the number of notifications received by users for all of your users, you see a power law like distribution where there are two major categories of users. There are users

like me who get a handful of notifications on a given day, and then there are these heavy users. They receive a ton of engagement. These tend to be your celebrities, your politicians, news reporters, and people who have gone viral. So as a product, this creates challenges

that have bearing on the infrastructure. So for instance, for these heavy users, they they get tons of engagement, so they get kind of notifications, and you want to make sure we deliver those to them in real time. On the other extreme, people who do not get a lot of notifications

are the ones who do not tweet a lot. They are the consumers, and for them the challenge is to figure out the right content that or right notifications that will bring value to their Twitter experience. As all of you are aware, Twitter is a very spiky product. Here's a

good example of that, and this is from 2013, so it's slightly dated, but this was from in Japan. They were broadcasting an anime called Castle in the Sky, and something must have happened during that where suddenly all the Japanese users took to Twitter and started tweeting it

and it created a tweets per second record for us back then, and then we started receiving hundreds of thousands of notifications or tweets per second. And as you can imagine, every time there's a tweet spike, there is a corresponding notification spike related to that.  Footer is highly asymmetric.

unlike some other social networks you have cases where people like Elon Musk who have millions of followers so every time Elon Musk tweets we have to send out his tweet to all of his followers and just like to expand out there's notifications pan out so every time we

tweets we have to find out all the users who we need to notify and then deliver that notification to them in real time so to round off an overview of our challenges how I'll briefly mention the four main challenges that we face and by no means this is an

exhaustive list there are a lot more both from product and infrastructure point of view but I want to highlight them today because we will also talk about how we solve or how we address some of these challenges latency is the most obvious one Twitter is live we want to make sure

that from the time someone tweets to the time you get notified is minimal we don't want to introduce much latency otherwise it takes away from the experience I spoke about notification spikes and find out again we have to make sure that we not only scale up to

handle billions of notifications every day but we can also handle these sudden spikes heterogeneity is something that's there for a lot of distributed systems and it's the case for notifications and what I mean there is if you look at all the calls that our services make

notification services make you can split them into two major categories there are there are calls that are serviced via cache or a fast data sort which tend to be anywhere from few milliseconds to let's say tens of milliseconds and then there are external calls that we make to

Google and iOS, which can take anywhere from half a second to a second. And when you have these two sets of calls, it's not as simple as just horizontally scaling up. You have to be cognizant of how you are coupling, or actually not coupling them. And then finally, for any

app like Twitter, we need to make sure that we are resilient. And for that, we need to make sure that our users can get their notifications even if we were to failover. And in general, we need to make sure that we can handle multiple data centers. So let's talk about how our

infrastructure addresses some of these challenges. And now, for the sake of time, I will be very, I will give you a very high-level overview. Each of this topic is something I can talk for the entire duration, but that's, give a quick overview. So this is our

push architecture, which is used to send push, SMS, and email notifications. And at a very high level, we see notifications, and then there is some business logic that we apply, things like make sure your settings are being honored, we check for spam abuse, whatnot. And so a

bunch of business rules are applied, and then once a notification passes all of them, it's then sent out for delivery, and then it eventually gets to your devices. So for this architecture, the, the three main challenges are, of course, we want to keep latency low because push, as I said,

is a live use case, live channel. We have to be able to deal with spikes, which happen all the time. And the problem, the spikes is, you cannot fit it when they happen. So I used to work at Netflix, and there they could rely on predictive auto

scaling, but we cannot do that because you have no idea when the next spike is going to come. And then I mentioned the heterogeneity, so how do we deal with latency and spikes? Well, we rely on horizontal scaling, but that's not enough. We need to make sure

that when we scale up or when we have a sudden surge of a spike or notifications traffic, we don't bring our upstream services down because, as I said, notifications are asymmetric and so it's very common for us to have millions of events in our queue that are all

referring to the same user. And if each of these processes start calling for the same user, that service, we will be down. So we rely on lots of short-lived caches. Usually, when you have you put caching in place, you optimize for cache hit rate. For us, it's totally fine to

have cache misses in the beginning, and given that these spikes are short-lived, short-lived caches totally suffice our use cases. In terms of heterogeneity, there are two main ways we deal with them. So, as I said, we have different types of notifications and different

types of calls, and we have, we rely heavily on priority queues to make sure that, for instance, your login codes are not being delayed because there's a backlog because Katy Perry just tweeted. So we need to decouple them, and we do that via priority queues. And then the

second thing is we make sure that tasks that have similar latency profiles are queued separately. So if there is a delay due to a Google outage or slowdown, we are not delaying other notifications. So that was the push architecture for notifications. Timeline, we use a pull architecture, and at a very

High-level, the same thing. Notifications are received via our right path. They're typically received asynchronously, but we have some synchronous use cases. We apply a set of business rules and, again, in this case, instead of pushing them out for delivery, we store them in a backing data store and they are cached and these

caches are long-lived caches. And the idea there is every time you visit your notifications timeline, it should be served out of cache. It's very rare that it's not served out of cache because it's a very costly operation to generate your timeline on the fly. So that's the two

main challenges here are again, latency, which is obvious. We want to make sure that we are not taking forever to load up your notifications timeline because otherwise you'll switch to something else. And the other thing is multi-DC. We need to make sure that in the event of

failure, failover, or whatnot, we can deliver your notification timeline to you and in a timely manner. So from a latency point of view, we use a custom implementation of Redis. And I'm not sure if you guys attended Yao's talk on Monday. She talked about our caching architecture. Very good

talk if you check it out. As I said, these we rely on long-lived caches and we also store all of our notifications in Manhattan, which is our real-time distributed backing store. And it's built for availability because that's the most important thing for us

from a multi-DC point of view. We, of course, cross the seer application, but that's not enough. We need to make sure that if you, you were to move from one data center to the other, your notifications stay the same. Otherwise, it will be a very jarring experience. And so

for that we have we siphon off a small volume of notifications and it's off in a separate asynchronous queue we make sure that we compare notifications across data centers and make sure that they are the same and if they are not the same we need to figure out what's

going on we also have maintenance jobs to make sure that our caches are not growing out of bounds and just doing housecleaning which we do not need to do while we are handling real-time events if you are building a notification infrastructure there these

two things I want to highlight which you should think about building in one is you want to make sure that it's very straightforward to add any notifications otherwise your experiment velocity will be very slow so most of our new notifications are pretty much self-serve or templated the second thing is we

want to ensure that your notifications and your experiments are driven purely on the backend side and the reason I mentioned this is as you can imagine the footer is on all possible apps and if I have to add a new notification and if I have to chase down the iOS team and an

Android team and make sure that I figure out when is their next release and they are never aligned and before before you know you have wasted months trying to add a new experiment and so we do not have we used to live in that world a long time ago and we have moved away

from that and most of our new experiments are now server-side and it is responsible for our high experiment velocity so from an infrastructure point of view before we wrap it up I want to mention these three key takeaways again there was a talk on Monday where this really good

talk where we talked about sinks, what is a sink, and that's a trade-off we make all the time. And for distributed systems, we should always rely on asynchronous operations. Asynchronicity is your friend. Try to make sure that you have as minimal synchronous operations as possible

because it's very hard to scale things up. So in our notifications infrastructure, we rely on very few synchronous operations and wherever possible, we try to move away from them. Similarly, we make trade-offs between write and read path, and the idea there is you need to be

cognizant of what you are doing at the right time. There are certain things you can do in real time, but you shouldn't, for instance, when we cache stuff, we only cache our IDs. We do not cache data for them because we want to do that at the

real time so that we give you the most up-to-date information. Similarly, on the flip side, when we show you an aggregated timeline, most of the aggregation happens at read time. There is no point doing it at write time. And then we talked about multi-DC, but and this stain

is you need to be very, even if you're starting off of an app with just one data center, make sure that you are aware of how you would scale it up to, or how you would evolve it to have multiple data centers. Now let's go over the

evolution of Twitter notifications. So we started off with this category of notifications which are by far the highest volume, they're called triggered notifications. As you can imagine, they are triggered by an interaction on Twitter. So for instance, if I were to tweet and if you were to reply or like

it will generate a like notification or reply notification so again these are the highest volume notifications by far as I mentioned we generate billions of them every day we need to make sure that not only we do that but we can handle spikes and I thought about

by modality in the beginning but trigger notifications are highly bimodal a small minority of users receive the majority of these notifications a quick example of how they flow in our system someone tweets goes through a right pass and then there we apply all our business

rules and logic and then once it passes that it's stored onto two different paths one path is real-time it leads to your device with a push and the other path is where it's stored and then served out of cache so why why do we need personalization why I triggered

notification is not enough the reason they are not enough is there is a lot more happening on Twitter both outside your graphs and within your graph that triggered notifications do not cover they cover a subset of those use cases but they are not enough for instance if

I'm a left-handed Warriors fan and I follow follow the Warriors account and then there is some controversy during a game and if the Warriors account doesn't tweet it I would never know about it but if Twitter knows that I'm interested in the Warriors they could notify me in real

time with a upcoming trend and whatnot as I mentioned in the beginning we notify users about what's happening in their world in real time we cover trigger notifications cover the real time part of it but there's a lot more that we do with personalization that

Gary will talk about so I'll hand it over to him. Thank you. So this is where I'm going to talk about personalization. Personalization comes down to notifying our users about their interests. You only want to hear about things you're interested in, and there's

two ways we do personalization at Twitter. The first we call personalized fan-out. Previously we talked about fan-out, when Elon Musk's tweets we have to send it to all the followers that have subscribed to them, but Elon Musk tweets about space because he's the CEO of SpaceX, and also tweets

about electric cars because he's CEO of Tesla. You might not be interested in space, but you might be interested in electric cars. So we only want to notify you when Elon Musk tweets about electric cars, and this is what personalized fan-out does. I'm going to

walk you through the personalized fan-out algorithm. So we mentioned we to keep track of a user's interest. We want to notify you about your interests, so we keep track of your recent engagements on entities. Let's break this down. Engagements are like tweet replies,

when you tweet about a particular topic, we call that an engagement, or when you like somebody else's tweet, we we bunch this up into something we call engagements. Entities, these are hashtags, accounts, or can be something more abstract like a particular topic. It could be movies, sports. So we keep track

of your engagement on these things, we call entities. To make this more concrete, for example, I'm currently in London, so I've tweeted about hashtag London. And this is being, I want to keep up with everybody's great talks, as in favoriting and liking and retweeting all

of your tweets about Q Khan, so I have recently engaged with London and Q Khan. So those are my recent engagements, so we keep track of these engagements as a proxy to what your interests are. And we only keep track of your recent engagements because Twitter is

constantly changing the data on Twitter. What people are talking about on Twitter are constantly changing, as well as your interests. You maybe haven't been interested in the Super Bowl or the Oscars a month or two months ago. You're not so interested in that right now, so

when we keep track of your most recent engagements, the second part to personalize find out is your followings. So when we mean followings, my followings are the users I follow. You follow a lot of people on Twitter, but not every single person you follow

means the same to you. There are some people that you really want to hear from, and there are others that you might not want to hear from them so much. So for every user, we want to keep track of who are the very, very top followings, who are

the people that they really, really want to hear, hear about. So for me, for example, I'm a Twitter employee, so Twitter is one of my top followings. And to come London, I want to hear about everybody's talks York on London. London tweets a lot about it, so

I follow maybe hundreds of people on Twitter, but in personal I span out, we would only use the top view. In this case, would take top two in the simplified example. So to recap, we have your recent engagements on entities, and we also have the top followings. And I want to walk

through an example of what happens when Q columns and tweets and when personalised inal decides whether they want to send it to me or not so first thing when you come across London tweets we extract out the entities from the tweet in this case we're looking at hashtags

so hashtag London is the entity we would then look whether I have recently engaged on hashtag London which I have because I'm at UConn in London and then we look at the author of the tweet and whether the author is in my top end followings so then we decide hey we'll

send this tweet to Khan Gary and the reason this is the hard problem like a lot of problems that Twitter is because there's asymmetric if you look at Katy Perry she has millions millions of followers to make it clear what we're not doing is looking at Katy Perry and

sending it to her top 20 followers because then there will be a huge fight to become the top 20 years of 2025 okay Perry and we upset a lot of people what we're doing is we're taking everybody that follows Katy Perry and seeing if

Katy Perry is in one of the top followings so you compare Katy Perry to someone slightly less popular like me I might only have one or two people who are in my talk who want to hear from me so how do we solve this so we solve this

by colocation and no network lookups okay what does let's deep dive into this what we do is we shard by every user so each user belongs to a particular shard I would be longing to say chars one star would be long into short too and on each

one of these charts for every user we keep track of the top engagement, their recent engagement and the top followings, and they're all on the same chart. What this means is when we want to run the personalized standout algorithm, we don't have to do network lookups to other

services. Everything's done locally, so we don't incur any latency by hitting other external services. They also scale well because we've charted the recent engagements and charted your followings, so we can scale horizontally that way. And Pokeno here is all of these charts

listen to the same copy of the firehose, so each one of these are running their personalized final algorithm on the users that belong to that shard. So let's walk through an example of what happens when Katy Perry actually tweets. So Katy Perry tweets here, 'I lost

the bet for best picture. I wanted glowing light.' So this was tweeted during the Oscars. If a lot of you might know, during the Oscars they announced that La La Land won best picture, and then they kind of, they made a mistake there and then they

announced that Moonlight won. So this Katy Perry tweeted this out before the fix, the mistake. But back to this problem, what we have supposing you live in a world where you have two shards, you have shard one and you have shard two, and

we're going to look at two of Katy Perry's followers, follower A and follower B, so far where A is not on shard one, and on shard one we keep track of follower A's engagements. In this case, follower A has engaged with La La Land, and Katy Perry is

one of the top followings on shore to follower B is a fan of moonlight and they've also followed Katy Perry's lemon cell followings so what happens in personalized fan out is again the first thing we do extract those entities from the tweet

in this case the entity is moonlight, a movie. We then send that tweet to every single personalized standout chart so all of these charts get it and they run the algorithm and as I mentioned before we look at the engagement only follower B has engaged with.  So we'd only

send this to follower B and not follower A. So in this way we're able to notify you about people you care about and only about the interests that you care about. So we talked about why we need to co-locate our data. I'm going to

deep dive now more into how we make this efficient and the key is data pre-processing. I'm going to go a bit backwards on this slide. I'm going to go from your right down to the left. So first off, the recent agents are entities we store that in memory. The reason we

store in memory is your recent agents are short-lived as I mentioned before. We keep you're not interested in something from two months ago, so we keep maybe a couple of hours up to a day's worth of your recent engagements in memory which is good.

We don't persist it as any for anywhere else because it's not useful anymore after a day or two. This data isn't really going to be useful, but the problem comes when your shard goes down. We work out of a data center. There's maintenance jobs, things go down all the time. All of you

are probably aware of this, and when you go down then you lose your in-memory recent engagements. At this point, we can't just go, "Oh sorry, we're not going to send you notifications", right? Because all the Katy Perry fans will get really angry. So what we do is we need to make

sure these personalized Dino shards come up very, very quickly, and the way we do it is we read from a queue. This queue has the last 24 hours of recent engagements, and on startup we've consumed from this queue very quickly to rebuild this in-memory

engagement graph. And there are two things that we do to make this efficient. The first is batching. By batching your engagements together, you reduce the message overhead, and this typically gives us a huge speed-up. The second thing we do is if you ever use the fire hose and you looked at the

Twitter JSON data, it contains a lot of metadata. So it has things like what clients the user used, what the iOS is, Android, which countries perhaps they tweeted from, what time of day the actual tweet pecks. There's all this metadata in there that aren't relevant for personal

life, I know. So we extract only the key pieces of metadata, for example, maybe just a user ID and the tweet ID that they've engaged with. So with that, that's why we call it a slim fire hose, and it's batched. So in this way, we're able to launch personalized, I

know, shards in a matter of minutes if they ever go down. And the other good thing about this is the entity extraction. So figuring out on the tweet which movies or hashtags it is sometimes can be quite an expensive process, and we do this into the slim engagement service. What this

means is we do the entity extraction once and then we send that result to all the personal life span out shards which means we save on that computation and you can put some more expensive computations in the entity extraction so this is recent engagements

the next is how do we pre-process the top followings so to find the top followings for every user we run an offline machine learning algorithm and this is based off your historical interaction between the two users for example whether you're likely to like that person's tweet whether you have

similar friends follow similar people on Twitter we calculate this offline because your friends and your top followings don't change that often friendships are built over months and years even in this modern age so running an offline allows us to make it reliable and scale correctly and after we've

calculated the top end followings for every single user we partition it and we partition it in the same way that we partition our production record service shards so if we have n rec service shards rent personalized I know shards sorry that service is what we call it

Real-Time & Personalized Notifications @Twitter

InfoQ - Human progress through technology

Software is changing the world, and our mission is to help progressive software development teams adopt new technologies and practices. For over 18 years, InfoQ has guided senior software development teams with peer-validated, trustworthy insights in adopting new technologies and practices.

InfoQ shares the experiences of senior software developers gained using innovator and early adopter stage techniques and technologies with the wider industry. We carefully curate and peer review everything we publish. We strongly believe that the high-quality insights offered by both our editors and other contributors have the power to uplift entire communities, no matter what their native language. 

InfoQ currently offers English, Japanese, Chinese, Portuguese, and French content.

Our conferences include QCon international software development conferences in London, New York, and San Francisco and the InfoQ Dev Summit in Boston and Munich.
