Product

News API Feature Update: Real-time monitoring

Introduction

With our News API, our goal is to make the world’s news content easier to collect, monitor and query, just like a database. We leverage Machine Learning and Natural Language Processing to process, normalize and analyze this content to make it easier for our users to gain access to rich and high quality metadata, and use powerful filtering capabilities that will ultimately help you to find precise and targeted stories with ease.

To this end, we have just launched a cool new feature, Real-time monitoring. Real-time monitoring allows you to further automate your collection and analysis of the world’s news content by creating tailored searches that source and automatically retrieve highly-relevant news stories, as soon as they are published.

real-time monitoring

You can read more about our latest feature – which is now also available in our News API SDKs – below.

Real-time monitoring

With Real-time monitoring enabled you can automatically pull stories as they are published, based on your specific search query. Users who rely on having access to the latest stories as soon as they are published, such as news aggregators and news app developers for example, should find this new feature particularly interesting.

The addition of this powerful new feature will help ensure that your app, webpage or news feed is bang up to date with the latest and most relevant news content, without the need for manual searching and updating.

Newly published stories can be pulled every minute (configurable), and duplicate stories in subsequent searches will be ignored. This ensures you are only getting the most recent publications, rather than a repeat of what has come before.

Usage

We have created code in seven different programming languages to help get you started with Real-time monitoring, each of which can be found below, as well as in our documentation.

require_once(__DIR__ . '/vendor/autoload.php');

function fetch_new_stories($opts){
  print_r("-----------\n");
  
  global $api_instance;
  
  $fetched_stories = [];
  $stories = NULL;
  
  while (is_null($stories) || count($stories) > 0) {
    try {
      $result = $api_instance->listStories($opts);
    } catch (Exception $e) {
      $code = $e->getResponseObject()->getErrors()[0]->getStatus();
      
      if ($code == '429'){
        print_r("Usage limit are exceeded. Wating for 60 seconds...\n");
        sleep(60);
        continue;
      }
    }
    
    $stories = $result->getStories();
    $fetched_stories = array_merge($fetched_stories, $stories);
    $opts['cursor'] = $result->getNextPageCursor();
    
    print_r(sprintf("Fetched %d stories. Total story count so far: %d\n",
      count($stories), count($fetched_stories)));
  }
  
  return $fetched_stories;
}


// Configure API key authorization: app_id
Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-ID', 'YOUR_APP_ID');

// Configure API key authorization: app_key
Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-Key', 'YOUR_APP_KEY');

$api_instance = new Aylien\NewsApi\Api\DefaultApi();

$params = array(
  'title' => 'Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM',
  'published_at_start' => 'NOW-1HOUR',
  'published_at_end' => 'NOW',
  'language' => ['en'],
  'per_page' => 5,
  'sort_by' => 'published_at',
  'sort_direction' => 'desc'
);

while(true){
  $stories = fetch_new_stories($params);

  print_r("************\n");
  print_r(sprintf("Fetched %d stories which were published between %s and %s\n",
    count($stories), $params['published_at_start'], $params['published_at_end']));
    
  if (count($stories) > 0){
    $lastFetchedAt = $stories[0]->getPublishedAt()->add(new DateInterval('PT10S'));
    
    $params['cursor'] = '*';
    $params['published_at_start'] = $lastFetchedAt->format('Y-m-d\TH:i:s\Z');
  }
  
  print_r("Sleep for 60 seconds until next poll...\n");
  print_r("-----------\n\n");
  
  sleep(60);
}
import time
import datetime
import aylien_news_api
from aylien_news_api.rest import ApiException

def fetch_new_stories(params={}):
  print('------------')
  
  fetched_stories = []
  stories = None
  
  while stories is None or len(stories) > 0:
    try:
      response = api_instance.list_stories(**params)
    except ApiException as e:
      if ( e.status == 429 ):
        print('Usage limit are exceeded. Wating for 60 seconds...')
        time.sleep(60)
        continue
        
    stories = response.stories
    params['cursor'] = response.next_page_cursor
    
    fetched_stories += stories
    print("Fetched %d stories. Total story count so far: %d" %
      (len(stories), len(fetched_stories)))
     
  return fetched_stories


# Configure API key authorization: app_id
aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
# Configure API key authorization: app_key
aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'

# create an instance of the API class
api_instance = aylien_news_api.DefaultApi()

params = {
  'title': 'Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM',
  'language': ['en'],
  'published_at_start': 'NOW-1HOUR',
  'published_at_end': 'NOW',
  'cursor': '*',
  'per_page': 5,
  'sort_by': 'published_at',
  'sort_direction': 'desc'
}

while True:
  stories = fetch_new_stories(params)
  
  print('************')
  print("Fetched %d stories which were published between %s and %s" %
    (len(stories), params['published_at_start'], params['published_at_end']))
  
  if len(stories) > 0:
    last_fetched_at = stories[0].published_at + datetime.timedelta(seconds=1)
    params['published_at_start'] = last_fetched_at.isoformat()[:-6] + 'Z'
    params['cursor'] = '*'
    
  print('Sleep for 60 seconds until next poll...')
  print('------------')
  time.sleep(60)
import com.aylien.newsapi.ApiClient;
import com.aylien.newsapi.ApiException;
import com.aylien.newsapi.Configuration;
import com.aylien.newsapi.api.DefaultApi;
import com.aylien.newsapi.auth.ApiKeyAuth;
import com.aylien.newsapi.models.Stories;
import com.aylien.newsapi.models.Story;
import com.aylien.newsapi.parameters.StoriesParams;
import org.joda.time.DateTime;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        // Configure API key authorization: app_id
        ApiKeyAuth app_id = (ApiKeyAuth) defaultClient.getAuthentication("app_id");
        app_id.setApiKey("YOUR_APP_ID");

        // Configure API key authorization: app_key
        ApiKeyAuth app_key = (ApiKeyAuth) defaultClient.getAuthentication("app_key");
        app_key.setApiKey("YOUR_APP_KEY");

        DefaultApi apiInstance = new DefaultApi();

        StoriesParams.Builder storiesBuilder = StoriesParams.newBuilder();

        storiesBuilder.setTitle("Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM");
        storiesBuilder.setLanguage(Arrays.asList("en"));
        storiesBuilder.setPublishedAtStart("NOW-1HOUR");
        storiesBuilder.setPublishedAtEnd("NOW");
        storiesBuilder.setCursor("*");
        storiesBuilder.setPerPage(5);
        storiesBuilder.setSortBy("published_at");
        storiesBuilder.setSortDirection("desc");

        while (true) {
            StoriesParams storiesParams = storiesBuilder.build();

            List stories = fetchNewStories(apiInstance, storiesBuilder);

            System.out.println("*****************");
            System.out.format("Fetched %d stories which were published between %s and %s\n",
                    stories.size(), storiesParams.getPublishedAtStart(), storiesParams.getPublishedAtEnd());

            if (stories.size() > 0) {
                DateTime lastFetchedAt = stories.get(0).getPublishedAt().plusSeconds(1);
                storiesBuilder.setPublishedAtStart(lastFetchedAt.toDateTimeISO().toString());
                storiesBuilder.setCursor("*");
            }

            System.out.println("Sleep for 60 seconds until next poll...");
            System.out.println("--------------");
            try {
                Thread.sleep(1000 * 60);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }

    public static List fetchNewStories(DefaultApi apiInstance, StoriesParams.Builder storiesParams) {
        System.out.println("--------------");

        List fetchedStories = new ArrayList();
        List stories = null;

        while (stories == null || stories.size() > 0) {
            try {
                Stories storiesResponse = apiInstance.listStories(storiesParams.build());
                stories = storiesResponse.getStories();
                storiesParams.setCursor(storiesResponse.getNextPageCursor());

                fetchedStories.addAll(stories);

                System.out.format("Fetched %d stories. Total story count so far: %d\n",
                        stories.size(), fetchedStories.size());
            } catch (ApiException e) {
                if (e.getCode() == 429) {
                    System.out.println("Usage limit are exceeded. Wating for 60 seconds...");
                    try {
                        Thread.sleep(1000 * 60);
                    } catch (InterruptedException ex) {
                        ex.printStackTrace();
                    }
                    continue;
                }
            }
        }

        return fetchedStories;
    }
}
# Load the gem
require 'aylien_news_api'
require 'time'

def fetch_new_stories(params={})
  puts "------------"
  
  fetched_stories = []
  stories = nil
  
  while stories.nil? || stories.size > 0
    begin
      result = $api_instance.list_stories(params)
    rescue AylienNewsApi::ApiError => e
      if e.code == 429
        puts 'Usage limit are exceeded. Wating for 60 seconds...'
        sleep(60)
        retry
      end
    end
    
    stories = result.stories
    params[:cursor] = result.next_page_cursor
    
    fetched_stories += stories
    puts "Fetched #{stories.size} stories. Total story count so far: #{fetched_stories.size}"
  end
  
  fetched_stories
end
  
# Setup authorization
AylienNewsApi.configure do |config|
  # Configure API key authorization: app_id
  config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID'
  
  # Configure API key authorization: app_key
  config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY'
end

# create an instance of the API class
$api_instance = AylienNewsApi::DefaultApi.new

opts = {
  :title => "Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM",
  :language => ['en'],
  :published_at_start => 'NOW-1HOUR',
  :published_at_end => 'NOW',
  :cursor => '*',
  :per_page => 5,
  :sort_by => "published_at",
  :sort_direction => "desc"
}

while true do
  stories = fetch_new_stories(opts)

  puts "*"*80
  puts "Fetched #{stories.size} stories which "\
    "were published between #{opts[:published_at_start]} and #{opts[:published_at_end]}"
    
  if stories.size > 0
    last_fetched_at = stories[0].published_at + Rational(1, 86400)
    opts[:published_at_start] = last_fetched_at.to_time.utc.iso8601
    opts[:cursor] = "*"
  end
  
  puts "Sleep for 60 seconds until next poll..."
  puts "------------\n\n"
  
  sleep(60)
end
package main

// Import the library
import (
	"fmt"
	newsapi "github.com/AYLIEN/aylien_newsapi_go"
	"time"
)

func main() {
	api := newsapi.NewDefaultApi()

	// Configure API key authorization: app_id
	api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID"

	// Configure API key authorization: app_key
	api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_APP_KEY"

	storiesParams := &newsapi.StoriesParams{
		Title:            "Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM",
		Language:         []string{"en"},
		PublishedAtStart: "NOW-1HOUR",
		PublishedAtEnd:   "NOW",
		Cursor:           "*",
		PerPage:          5,
		SortBy:           "published_at",
		SortDirection:    "desc"}

	for {
		stories := fetchNewStories(api, storiesParams)

		fmt.Println("***************")
		fmt.Printf("Fetched %d stories which were published between %s and %s\n",
			len(stories), storiesParams.PublishedAtStart, storiesParams.PublishedAtEnd)

		if len(stories) > 0 {
			lastFetchedAt := stories[0].PublishedAt.Add(time.Duration(1) * time.Second)
			storiesParams.PublishedAtStart = lastFetchedAt.Format("2006-01-02T15:04:05Z")
			storiesParams.Cursor = "*"
		}

		fmt.Println("Sleep for 60 seconds until next poll...")
		fmt.Println("------------")
		time.Sleep(time.Duration(60) * time.Second)
	}
}

func fetchNewStories(apiInstance *newsapi.DefaultApi, params *newsapi.StoriesParams) []newsapi.Story {
	fmt.Println("---------------")

	var fetchedStories []newsapi.Story
	var stories []newsapi.Story
	isFirstCall := true

	for isFirstCall || len(stories) > 0 {
		storiesResponse, res, err := apiInstance.ListStories(params)
		if err != nil {
			panic(err)
		}

		if res.Response.StatusCode == 429 {
			fmt.Println("Usage limit are exceeded. Wating for 60 seconds...")
			time.Sleep(time.Duration(60) * time.Second)
			continue
		}

		isFirstCall = false
		stories = storiesResponse.Stories
		params.Cursor = storiesResponse.NextPageCursor

		fetchedStories = append(fetchedStories, stories...)

		fmt.Printf("Fetched %d stories. Total story count so far: %d\n", len(stories), len(fetchedStories))
	}

	return fetchedStories
}
var AylienNewsApi = require('aylien-news-api');
var Promise = require('bluebird');

// Async Promise “while loop”
function promiseWhile(condition, action) {
  var resolver = Promise.defer();

  var loop = function() {
    if (!condition()) return resolver.resolve();
    return Promise.cast(action())
      .then(loop)
      .catch(resolver.reject);
  };

  process.nextTick(loop);

  return resolver.promise;
}

// Async Promise “sleep”
function sleep(time){
  return new Promise((resolve) => setTimeout(resolve, time));
}

function fetchNewStories(opts){
  console.log('-------------');
  
  return new Promise(function (resolve, reject){
    var fetchedStories = [];
    var stories = null;
    var whileCondition = function(){
      return (stories == null || stories.length > 0);
    };
    
    promiseWhile(whileCondition, function(){
      return new Promise(function (resolve, reject){
        apiInstance.listStories(opts, function(error, data, response){
          if (error){
            if (response.status == 429) {
              console.log('Usage limit are exceeded. Wating for 60 seconds...');
              sleep(60 * 1000).then(function(){
                resolve();
              });
            } else {
              reject(error);
            }
          } else {
            opts.cursor = data.nextPageCursor;
            stories = data.stories;
            
            fetchedStories = fetchedStories.concat(stories);
            
            console.log("Fetched " + stories.length +
              " stories. Total story count so far: " + fetchedStories.length);
            
            resolve();
          }
        });
      });
    }).then(function(){
      resolve(fetchedStories);
    });
  });
}


var apiInstance = new AylienNewsApi.DefaultApi();

// Configure API key authorization: app_id
var app_id = apiInstance.apiClient.authentications['app_id'];
app_id.apiKey = "YOUR_APP_ID";

// Configure API key authorization: app_key
var app_key = apiInstance.apiClient.authentications['app_key'];
app_key.apiKey = "YOUR_APP_KEY";

var opts = {
  'title': 'Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM',
  'language': ['en'],
  'publishedAtStart': 'NOW-1HOUR',
  'publishedAtEnd': 'NOW',
  'cursor': '*',
  'perPage': 5,
  'sortBy': 'published_at',
  'sortDirection': 'desc'
};
    
var whileCondition = function(){ return true; };

promiseWhile(whileCondition, function(){
  return new Promise(function (resolve, reject){
    fetchNewStories(opts).then(function(stories){
      console.log('**************');
      console.log('Fetched ' + stories.length +
        ' stories which were published between ' + opts['publishedAtStart'] +
        ' and ' + opts['publishedAtEnd']);
        
      if (stories.length > 0) {
        var newFetchAt = stories[0].publishedAt.getTime() + 1000;
        opts['publishedAtStart'] = new Date(newFetchAt);
        opts['cursor'] = '*';
      }
      
      console.log("Sleep for 60 seconds until next poll...");
      console.log('-------------\n');
      sleep(60 * 1000).then(function(){
        resolve();
      });
    });
  });
});
using System;
using Aylien.NewsApi.Api;
using Aylien.NewsApi.Client;
using Aylien.NewsApi.Model;
using System.Collections.Generic;

namespace PollingStories
{
    class Program
    {
        static DefaultApi apiInstance;

        static void Main(string[] args)
        {
            // Configure API key authorization: app_id
            Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-ID", "YOUR_APP_ID");

            // Configure API key authorization: app_key
            Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-Key", "YOUR_APP_KEY");

            apiInstance = new DefaultApi();
            var publishedAtStart = "NOW-1HOUR";
            var publishedAtEnd = "NOW";

            while (true)
            {
                var stories = FetchNewStories(publishedAtStart, publishedAtEnd);

                Console.WriteLine("*****************");
                Console.WriteLine(string.Format("Fetched {0} stories which were published between {1} and {2}",
                    stories.Count, publishedAtStart, publishedAtEnd));

                if (stories.Count > 0)
                {
                    var lastFetchedAt = stories[0].PublishedAt.Value.AddSeconds(1);
                    publishedAtStart = lastFetchedAt.ToString("yyyy-MM-ddTHH:mm:ssZ");
                }

                Console.WriteLine("Sleep for 60 seconds until next poll...");
                Console.WriteLine("------------");
                System.Threading.Thread.Sleep(1000 * 60);
            }
        }

        static List FetchNewStories(String publishedAtStart, String publishedAtEnd)
        {
            Console.WriteLine("--------------------");

            var cursor = "*";
            var fetchedStories = new List();
            List stories = null;

            while (stories == null || stories.Count > 0)
            {
                try
                {
                    var storiesResponse = apiInstance.ListStoriesWithHttpInfo(
                        title: "Apple OR Amazon OR Facebook OR PayPal OR Microsoft OR Ebay OR Cisco OR Netflix OR IBM",
                        language: new List { "en" },
                        publishedAtStart: publishedAtStart,
                        publishedAtEnd: publishedAtEnd,
                        cursor: cursor,
                        perPage: 5,
                        sortBy: "published_at",
                        sortDirection: "desc"
                    );


                    stories = storiesResponse.Data._Stories;
                    cursor = storiesResponse.Data.NextPageCursor;
                    fetchedStories.AddRange(stories);

                    Console.WriteLine(string.Format("Fetched {0} stories. Total story count so far: {1}",
                        stories.Count, fetchedStories.Count));
                }
                catch (ApiException e)
                {
                    if (e.ErrorCode == 429)
                    {
                        Console.WriteLine("Usage limit are exceeded. Wating for 60 seconds...");
                        System.Threading.Thread.Sleep(1000 * 60);
                    }
                    else
                    {
                        Console.WriteLine("Exception when calling DefaultApi.ListStories: " + e.Message);
                    }

                }
            }
            return fetchedStories;
        }
    }
}

NB: Real-time monitoring will only work when you set the sort_by parameter to published_at and sort_direction to desc.

Benefit

The main benefit of this cool new feature is that you can be confident you are receiving the very latest stories and insights, without delay, by creating an automated process that will continue to retrieve relevant content as soon as it is published online. By automating the retrieval of content in real-time, you can cut down on manual input and generate feeds, charts and graphs that will automatically update in real-time.

We hope that you find this new update useful, and we would love to hear any feedback you may have.

To start using our News API for free and query the world’s news content easily, click the image below.

 


News API - Sign up




Author


Avatar

Noel Bambrick

Customer Success Manager @ AYLIEN A graduate of the Dublin Institute of Technology and Digital Marketing Institute in Ireland, Noel heads up Customer Success here at AYLIEN. A keen runner, writer and traveller, Noel joined the team having previously gained experience with SaaS companies in Australia and Canada. Twitter: @noelbambrick