Behind the Scenes: Delivering Real-Time News to Apps

In today’s fast-paced digital world, delivering news in real-time has become a fundamental expectation for users. This is especially true in the context of sports apps, where up-to-the-minute updates can make or break the user experience. But what does it take to build and maintain a system capable of delivering seamless news updates, push notifications, and personalized experiences to both iPhone and Android apps?

This blog takes you behind the scenes of a recent project where we tackled this challenge head-on. From implementing AI-powered tagging to managing a distributed API capable of handling 100K+ requests per hour, here’s how we approached the complexities of delivering real-time news at scale.

Step 1: Enhancing the Editorial Process with NER

Before news even reaches the database, the editorial process ensures that every article is enriched with metadata and tags. To achieve this, we developed a NER (Named Entity Recognition) API that scans articles in real-time as editors work on them. The API identifies:

• Player and team names.
• Locations and places.
• Relevant entities like competitions or clubs.

With a simple click, editors can select accurate tags, often better than they might choose manually. This system not only accelerates the tagging process but also ensures uniformity. Additionally, tags can automatically link players, teams, or competitions to detailed pages in the apps, creating a richer user experience.

Step 2: From Publication to Website

Once an article is ready, it enters the Galera database cluster. Galera’s high availability ensures that articles are redundantly stored and immediately accessible. A cluster of web servers behind load balancers retrieves these articles and displays them on the website.

However, the story doesn’t stop at the website. The same information must also be prepared for distribution to iPhone and Android apps, requiring a separate, high-performance API.

Step 3: Building a High-Performance API

Developing an API to handle news delivery and push notifications comes with its own set of challenges:

1. High Request Volume:
   With apps supporting push notifications for many teams and many thousands users consuming news in real-time, the API had to handle 100K+ requests per hour.
2. Efficient Query Design:
   Every SQL query needed careful thought. A single inefficient query could multiply into millions of extra requests, significantly impacting performance and costs.
3. Caching for Scalability:
   To balance speed and efficiency, we implemented a Redis cache, where each web server is part of a Redis cluster. This ensures scalability, redundancy, and local access to cached data. Cache times are fine-tuned based on the freshness requirements of different data types.

Step 4: Monitoring and Fine-Tuning

Building a scalable API isn’t just about coding—it’s about anticipating and mitigating issues during live operations. This is where monitoring played a crucial role:

• Error Tracking:
  Using tools like Grafana and our monitoring system, we tracked critical metrics such as 404 and 503 HTTP error rates and monitored for full message queues.
• Queue Monitoring:
  Our monitoring system counted records in queues every minute, providing insights into potential bottlenecks before they became critical.
• Load Testing:
  Extensive testing ensured the system could handle peak traffic without failures. This included tuning Galera’s optimistic locking to minimize deadlocks under high concurrency and experimenting with techniques like splitting counters into 100 buckets to avoid bottlenecks.

Step 5: Real-Time Push Notifications

With the API in place, the next challenge was delivering push notifications. We integrated Firebase Cloud Messaging (FCM) to manage push subscriptions for teams and ensure notifications reached users instantly. This system:

• Processed 800-2000 registrations and subscriptions per minute without any errors post-deployment.
• Allowed users to subscribe to specific team notifications, ensuring personalized updates.

Step 6: Deployment and Results

Deploying such a complex system always comes with risks. Despite the potential for errors, the system went live with only one minor hiccup—a few images failed to load initially. Other than that, the launch was flawless. The system now seamlessly delivers real-time news to apps, handles push notifications efficiently, and scales effortlessly to meet growing demand.

Lessons Learned

1. Efficiency is Everything:
   Building an API for high traffic requires meticulous query design, caching strategies, and concurrency handling.
2. Monitoring Saves the Day:
   Real-time monitoring and visualization through Grafana provided critical insights during deployment and beyond.
3. Iterate and Experiment:
   Solutions like splitting counters into buckets or fine-tuning cache times only emerged after extensive testing and experimentation.
4. AI Can Simplify the Complex:
   The NER API demonstrated how AI tools can make editorial processes faster, more accurate, and more uniform.

Looking Ahead

This project was a prime example of how back-end infrastructure, APIs, and front-end interfaces must work in harmony to deliver exceptional user experiences. Moving forward, we plan to:

• Explore more advanced caching strategies.
• Enhance push notifications with real-time analytics.
• Integrate more AI-driven features to personalize content delivery further.

Delivering real-time news to apps might seem simple to the end user, but as this story shows, it’s a symphony of carefully orchestrated systems and processes.