Notification System Design Interview Walkthrough

The first decision in a notification system design is to keep urgent transactional messages, such as one-time passwords (OTPs) and payment alerts, apart from bulk promotional campaigns, with separate queues, separate workers and a reserved share of each provider's rate limit. Duplicates, retries and campaign speed all depend on the providers, because your system hands every message to Firebase Cloud Messaging (FCM), the Apple Push Notification service (APNs), an email service or an SMS gateway, and each has its own quota, error codes and delivery rules.

Requirements

Ask who calls the system (internal services through an API, a marketing tool that targets a segment, or both), which channels are in scope, which countries the users are in, and what the business means by "delivered". Then split the traffic into two types, because most later decisions differ between them.

If the interviewer gives a scale, use it. If not, state your own numbers and say they are assumptions.

Rough numbers

Assume 20 million daily active users, each getting 5 transactional notifications a day.

• 20,000,000 × 5 = 100,000,000 transactional notifications a day.
• 100,000,000 ÷ 86,400 seconds ≈ 1,160 a second on average.
• With peak at 3 times the average, that is about 3,500 a second, or about 210,000 a minute.

Now add one promotional push to all 20 million users. Firebase's throttling and quotas page (last updated 2026-10-07) sets a default quota of 600k messages a minute per project, and over quota FCM returns HTTP 429 until the quota refills. If campaigns and transactional push share one Firebase project, the campaign cannot take the whole quota. Cap it at 360,000 a minute (6,000 a second), which leaves 240,000 a minute for transactional push. That check is conservative, because it counts all 210,000 transactional messages a minute at peak as FCM push. The 30,000 a minute left over also has to cover retries and failed sends.

• 20,000,000 ÷ 360,000 a minute ≈ 56 minutes for the campaign.
• With the whole 600,000 a minute and nothing else running, 20,000,000 ÷ 600,000 ≈ 33 minutes.
• Finishing in 10 minutes needs 2,000,000 a minute, more than three times the default quota.

The 10-minute version needs more than the default quota. On the same Firebase page, a self-serve increase is capped at 25% (750,000 a minute here) and is open only to projects that regularly use 80% of their quota with under 5% client errors. Anything bigger is a request to Firebase Support made at least 15 days before the launch. In the interview, propose a campaign window of about an hour and name the quota as the reason.

Keep a record of every send for status lookups and support questions. At 120 million sends a day (100 million transactional plus the 20 million campaign) and an assumed 500 bytes a record, that is 120,000,000 × 500 bytes = 60 GB a day, or 5.4 TB over 90 days. Peak inserts are about 3,500 + 6,000 = 9,500 a second before any status updates. Partition the table by day so that old data goes by dropping a partition.

Twilio bills SMS per segment. A GSM-7 message fits 160 characters in one segment, but any character from a non-Latin script switches the whole message to UCS-2, which fits 70 characters, or 67 a segment once split. A 90-character OTP message in English is one segment. The same message in Hindi, written in Devanagari, is two segments and costs twice as much to send.

Queues and fan-out

The API validates a request, writes a notification record, puts a message on a queue and returns. Make the record and the enqueue succeed or fail together, for example with a transactional outbox, where the record and an outbox row commit in one database transaction and a relay publishes outbox rows to the queue. Without that, a crash between the two steps leaves a record with no message, or a message with no record.

Give each priority and channel its own queue and worker pool, such as transactional push, transactional SMS, transactional email and promotional. With one shared queue, an OTP that arrives just after the campaign starts waits behind up to 20 million promotional messages, which at 360,000 a minute is close to an hour. Separate channel pools also mean a slow SMS gateway holds up only SMS.

A campaign arrives as one request naming a segment. A fan-out job reads the segment in pages, say 1,000 users at a time, and writes one message per user to the promotional queue. A shared token bucket set to 6,000 a second paces the workers. Firebase's scaling guide asks senders to do three things during a burst like this.

• Ramp from zero to full rate over at least 60 seconds.
• Avoid the two minutes around :00, :15, :30 and :45. FCM receives more than double its usual traffic in the first 30 seconds to 2 minutes of each hour, so a campaign set for exactly 9:00 lands on that spike.
• Spread sends evenly inside each minute. Firebase's quota minutes are not aligned to the clock, so a burst at the start of your own minute can still draw 429s.

Kafka keeps each partition in total order, according to its design documentation, so keying messages by user ID gives per-user order inside your system. That order can change at the provider. Apple's APNs request guide says APNs, as a best-effort service, may reorder notifications sent to the same device token, so on iPhones the display order is outside your control.

The queue can be Kafka or Amazon SQS, and three features decide between them. For per-user order, an SQS FIFO queue keeps strict order within one message group ID, which you would set to the user ID to match what a Kafka partition gives you. For replay after a bug, a Kafka consumer can rewind and re-consume messages, but those messages already carry "sent" records, so a replay needs a flag that lets chosen messages past the dedup check. For retries, SQS hides a received message for a visibility timeout (30 seconds by default, up to 12 hours), hands it out again if the worker does not delete it, and moves it to a dead-letter queue after maxReceiveCount receives. Pick Kafka when replay and per-user order matter most, and SQS when its built-in redelivery and dead-letter queue matter more.

Delivery guarantees and duplicates

Promise at-least-once delivery to the provider. Kafka guarantees at-least-once by default, and its design documentation says exactly-once delivery to other systems "generally requires cooperation with such systems". SQS standard queues are at-least-once too, and AWS tells you to design consumers to be idempotent. For push, the other system is FCM or APNs. Apple's docs say each request to APNs typically displays a new notification on the device, so a repeated send shows up twice.

Duplicates come from the gap between the provider accepting a message and your system recording it. A worker may crash right after a success response, or time out while the provider has in fact accepted the message. Either way the queue hands the message to another worker, which sends it again. A dedup key cannot close that gap, and where you write the key decides which failure you accept.

• Write "sent" after the provider call succeeds. A crash between the call and the write gives a duplicate, and nothing is lost.
• Claim the key before the call. A crash or a failed call after the claim leaves a key for a message that never went out, the retry skips it, and that message is lost.

Choose per type. For an OTP or a payment alert, a duplicate costs less than a loss, so record after the call. For a campaign, one lost promotion costs little and a duplicate annoys the user, so claim first. Store a state per notification (pending, sending with a lease time, sent). A redelivered message already marked sent is dropped, and one stuck in sending past its lease is the uncertain case that the type rule decides.

Push gives you one more guard on the device. Apple documents an apns-collapse-id header of up to 64 bytes; when you send the same notification more than once, the same value merges the requests into one notification. On Android, the FCM notification tag replaces a notification already shown with the same tag. Set both to the notification ID, and a retry that gets through replaces the first notification on screen. For email and SMS, your sent record is the guard.

Keep each dedup key only as long as a duplicate could arrive. An OTP's key can expire with the OTP. A campaign's key has to outlast the campaign and its retries, so about a day. A day of keys at 120 million sends and about 100 bytes a key is 12 GB, which fits in memory on a small Redis cluster.

Preferences, quiet hours and caps

Check preferences in the worker, just before the provider call. A campaign runs for about an hour, and a user who opts out at minute 10 should not get it at minute 50. Cache preferences with a short expiry and clear a user's cache entry when they change a setting. For each message the worker asks, in order:

1. Is this channel enabled for this user and this message type?
2. Is the user in quiet hours? This applies to promotional messages only, and the message waits until the user's morning in their own time zone.
3. Has the user reached their daily cap? The cap is a product choice, for example 3 promotional pushes a day.
4. Is there a valid device token or a verified address?

For SMS to Indian users through Twilio's domestic route, Twilio's India guidelines require the company and sender ID to be registered on the mobile operators' DLT portal first. The same page asks senders to message users only during their daytime hours unless it is urgent, and not to contact users on do-not-disturb registries.

For email, Google's sender guidelines require anyone sending more than 5,000 messages a day to Gmail accounts to support one-click unsubscribe on marketing messages and to keep the spam rate in Postmaster Tools below 0.3%. The unsubscribe endpoint has to write to the same preference store the workers read, or the next campaign reaches someone who left.

Remove push tokens when the provider says they are dead. FCM returns UNREGISTERED (HTTP 404) when a token is no longer valid. APNs returns 410 with the reason Unregistered or ExpiredToken, and the response includes the time APNs confirmed the token was no longer valid. Firebase also expires Android registrations after 270 days of inactivity. Dead tokens cost quota. Firebase counts client errors (HTTP 400 to 499, except 429) against the quota, so every send to a dead token uses quota, and those 404s count toward the client error ratio.

Retries and provider errors

Read the error before you retry. Each provider documents which errors are worth another attempt.

These rules come from Firebase's scaling guide and Apple's response codes guide. Email has its own case. Amazon SES drops a message it throttles and does not try again, so your worker owns the retry, and AWS's sending errors page says to wait up to 10 minutes before sending again. A wait of up to 10 minutes can use up the whole life of a 10-minute OTP, which is one more reason to send OTPs by SMS or push.

Retry in one place. AWS's Builders' Library gives the example of a five-layer stack with three retries at each layer, where a failing database at the bottom sees 243 times the load. For low-cost control-plane and data-plane operations, AWS's general practice is to retry at a single point in the stack. Here that point is the delivery worker and its queue, and the HTTP client inside the worker does not retry on its own. With SQS, each receive is one attempt, and the worker can change the visibility timeout of a failed message to the wait the provider asks for. With Kafka, the worker runs the backoff itself or moves the message to a separate retry topic.

Every message also carries a deadline, and the worker checks it before each attempt. Assume an OTP is valid for 10 minutes, then pass that deadline to each provider's expiry field.

• FCM ttl is how long FCM keeps a message for an offline device. The default and the maximum are both 4 weeks, so without it an OTP to a phone that was off can arrive weeks later.
• APNs apns-expiration is the time after which the notification is no longer valid, and 0 means APNs tries once and does not store it. Otherwise APNs stores a notification for up to 30 days, keeping only one per app on the device, usually the latest. Apple honours the expiry on a best-effort basis, so a notification can still arrive after it.
• Twilio validity_period is how long a message may wait in Twilio's outgoing queue, from 1 to 36,000 seconds, and the default is 36,000 (10 hours).

When the deadline passes, drop the message and count the drop. A dead-letter queue holds messages someone will inspect or replay, and an expired OTP has no use to anyone. Promotional messages that keep failing go to the dead-letter queue.

What to measure

Measure each channel and message type separately.

• Latency from request to provider acceptance for transactional messages, at the 99th percentile.
• Queue age, the age of the oldest message in each transactional queue. It rises before latency does, so alert on it first.
• Accepted and delivered as separate counts. A provider's success response means it accepted the message. Twilio reports "sent" when the nearest carrier accepts a message and "delivered" when the carrier confirms delivery. Amazon SES publishes separate Send, Delivery, Bounce, Complaint, Reject and DeliveryDelay events. Apple's docs say an accepted push may still be throttled, held in storage or, in some cases, not delivered.
• Error mix by code. Dead-token errors (FCM 404, APNs 410) track token cleanup, 429s track quota and pacing, and 5xx errors track provider health.
• Expired drops for each message type.
• Email spam rate against Gmail's limits, and SMS segments per message for each template language.

Practising these decisions

CramHQ's free System Design assessment takes about 15 minutes and needs no credit card. It asks system design scenario questions, then shows which design decision areas are weakest. CramHQ's paid Pass Plan costs USD 49.99 once for 12 months of access and adds targeted repair work, a Readiness Exam and the 16 Decision Drills listed on the course page, including drills on write path and async processing, reliability, and observability.