Product

Feature Update: Related Phrases Added to the Text Analysis API

Our most recent feature addition to the API, Related Phrases, automatically provides you with a list of semantically similar phrases and words based on an input phrase or word provided by the user. Put simply, given a phrase made up of as many words as you like (unigram, bigram or n-gram) our API will return a list of words or phrases, that are similar or related.

So how does it work?

Below is an example of Related Phrases in action, using “iPhone” as the example phrase. The screenshot from our demo below shows the returned results, along with a distance score. The distance score indicates how similar the context of the suggested words/phrases is to one that the source words/phrases might appear in, and the higher the distance score the more closely related they are.

In other words we build a profile/signature for each word based on its surrounding words in a corpus (English Wikipedia for instance). The profile we build indicates, among other things, how likely each word is to appear with any/all other words in the corpus.

Example:

 

 

So we’ve looked at a simple example using “iPhone” as our input, but we’ve also built a cool little sample app in our Sandbox which illustrates a neat little idea we had, of how the feature could be used.

The idea for the sample app was to take a story or even a news headline and change it up automatically without affecting its meaning, with the possibility of sometimes improving its word structure. We do this by extracting Keywords and Entities from a sample string and swapping out one of those keywords or entities and replacing them with related words or phrases.

We managed to get some pretty interesting results, which we’ve displayed below.

Note: We’ve included the code below for you to copy and paste or even add to build something cooler. Just remember to swap out your App ID and App Key which you would have received when you registered with AYLIEN. If you haven’t registered, you can do so here.

Code Snippet:

var AYLIENTextAPI = require('aylien_textapi'),
  _ = require('underscore'),
  request = require('request'),
  parseString = require('xml2js').parseString;

var textapi = new AYLIENTextAPI({
  application_id: 'YourApplicationId',
  application_key: 'YourApplicationKey'
});

request('http://feeds.bbci.co.uk/news/rss.xml', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    parseString(body, function(error, feed) {
      var story = _.sample(feed.rss['channel'][0].item).description[0] || undefined;
      if (story) {
        textapi.entities(story, function(error, entities) {
          var keyword = entities.entities.keyword[0];
          textapi.related(keyword, function(error, related) {
            var random_phrase = _.sample(related.related.slice(0,5));
            console.log("Results:n");
            console.log("- Story: " + story);
            console.log("- Selected keyword: " + keyword);
            console.log("- Related phrases: " + _(related.related.slice(0,5)).pluck('phrase').join(', '));
            console.log("- New Story: " + story.replace(new RegExp(keyword,"g"), '*'+random_phrase.phrase+'*'));
          });
        });
      }
    });
  }
});

As you can see from the code, we’re grabbing a story from the BBC top stories feed and running the following analyzes on it to determine what words to replace.

 

  • Entity Extraction
  • Keyword Extraction

 

Then we’re choosing the most prominent keyword and passing that through the Related Phrases endpoint, to generate our auto-generated phrase or words. Choosing one phrase or word at random we replace it in the original “story” to essentially rewrite a part of that article snippet, as is illustrated in the example results below.

Sample App Results:

- Story: Climate change risks should be assessed in the same way as threats to national security according to a new report.
- Selected keyword: threats
- Related phrases: threat, threatening, potential threats, serious threats, threaten
- New Story: Climate change risks should be assessed in the same way as *serious threats* to national security according to a new report.
- Story: A human rights group publishes a video it says contradicts the account of an Israeli army officer who shot dead a Palestinian teenager a week ago.
- Selected keyword: Israeli
- Related phrases: israel, idf, israelis, israel defense forces, palestinian
- New Story: A human rights group publishes a video it says contradicts the account of an *israel defense forces* army officer who shot dead a Palestinian teenager a week ago.
- Story: A city council is opposing a request to hand over an area of land for the site of a new free Sikh school in Derby.
- Selected keyword: land
- Related phrases: lands, large tracts, large tract, property, acres
- New Story: A city council is opposing a request to hand over an area of *property* for the site of a new free Sikh school in Derby.

Other Use Cases

Cool huh? Even cooler than that, we thought would it be possible to rewrite an entire passage or article by replacing keywords and entities with other related words and phrases without losing meaning and readability. We haven’t got around to building that yet, but there’s free swag on offer for anyone who can do it! ;).

There are some pretty cool and quirky ideas out there for how this endpoint could be used. One of the most useful ideas we’ve had was, to use the Related Phrases end point as part of, the keyword research process in SEO/search marketing. This process proves very useful in generating long tail phrases and semantically related search terms, which are becoming more and more important in Google’s eyes as it moves away from keyword spotting. Keep an eye out for a blog on that very use case in the coming days.

Related Phrases is now live in production and can be used by all of our API subscribers, both free and paid. If you want to try it out right now check out our demo and sandbox or read about it in our docs.





Text Analysis API - Sign up




Author


Avatar

Mike Waldron

Head of Marketing & Sales @ AYLIEN A legal convert with a masters degree from Smurfit Business School, Mike runs our Sales and Marketing at AYLIEN. Mike gathered his Sales and Marketing experience with technology companies in Sydney and Dublin before getting the startup itch and joining the team at AYLIEN. Twitter: @MikeWallly