Product

Sentiment Analysis of Tweets with AYLIEN Text Analysis API

At AYLIEN, we do our best to make sure our users get up and running and calling our API in the shortest time possible. As part of a new initiative, we are going to be sharing use case ideas, source code and fully functional apps to help you get the most out of our API. For this edition of the blog, we are going to focus on a pretty common use case that a lot of users want to use our API for, analyzing Tweets.

There is a wealth of insight that can be extracted from Tweets. You can read more on Analyzing Tweets and Social data in our previous blog; Why is Sentiment Analysis important from a business perspective.

Today, we’re going to provide you with the source code for a functioning app that mines Twitter for keywords, extracts Tweets and analyzes the text in the Tweets. As part of the process, we’ll run two analysis endpoints on each Tweet, Sentiment Analysis on all of the Tweets and Hashtag Suggestion on Tweets that contain a URL.

What you’ll need to get going:

 

  • Twitter API access: Get your api_key, consumer_secret key and access token to make calls to the Twitter API here.
  • Node.js running on your machine: If you don’t already have Node.js on your machine you can download it here.
  • Twit: A Twitter API client library for Node.js. You can download it here.
  • A text editor: You can use any editor, We recommend Sublime Text and you can download it here.
  • AYLIEN Text Analysis API access: Get your Application ID and Application Key here. See our ‘getting started’ blog for details on how to sign up.

 

Overview of the code in action

To give an overview of what can be achieved, we will first look at the code in action. The complete code snippet is given at the end of this blog for you to copy and paste.

Step1. Setup your Environment

Ensure Node.js is running on your machine, download the twit client library from GitHub, get access to the Twitter API and finally, open an AYLIEN Text Analysis API account.

Step 2. Copy the Code

Open your text editor and copy and paste the code snippet (provided at the bottom of this blog) and save the file as, tweetsentiment.js. Next, open command prompt and Navigate to the folder where you saved the code snippet.

The application takes two command line parameters which you chose; a keyword for the Twitter query and the number of Tweets the query should return.

Step 3. Run your Code

Run the code by typing “tweetsentiment websummit 3”. In this case we are querying the keyword “websummit” and asking for 3 Tweets to be returned.

Once the Tweets are returned by the Twitter API they are fed to AYLIEN Text Analysis API, where the polarity will be determined and where the optimal Hashtags for URL’s will be generated.

Note: Ensure you replace the YOUR_APP_ID and YOUR_APP_KEY placeholders in the code with the application id and application key which you received when you signed up for the AYLIEN API. You will also need to fill in your specific Twitter API credentials that you received from Twitter. All going well you should see an output on the command line similar to that shown below.

Results:

Tweet Text           :  RT @IndoBusiness: The #WebSummit is drawing to a close. #Bono up soon. Watch live here: http://t.co/onpdYvoIy4 Or follow our blog here: htt.
Sentiment Polarity   :  neutral
Polarity Confidence  :  0.9702560119839743
Hashtags :  [
  '#RyderCup',
  '#PeterThiel',
  '#Davos',
  '#AdrianGrenier',
  '#PaulMcGinley',
  '#FoundersFund'
]
Tweet Text           :  RT @FierceClever: I swear to god, if I hear the word "di
srupt" one more time... #websummit
Sentiment Polarity   :  negative
Polarity Confidence  :  0.8947368421052632
Hashtags :  No Hastags available as no Url specified in the Tweet
Tweet Text           :  Having a great day at the #websummit . Were at stand ECM 243 in the village if anyone would like to pop over before closing!
Sentiment Polarity   :  positive
Polarity Confidence  :  0.9230769230769231
Hashtags :  No Hastags available as no Url specified in the Tweet

Taking the first result as an example, you can see that the Tweet itself is displayed followed by the “sentiment polarity” of the Tweet (positive, neutral or negative) and the “polarity confidence” i.e. the confidence that the sentiment returned was correct (a number between 0 and 1). Finally, if the Tweet contained a URL embedded in the Tweet a list of optimal hashtags is generated for that webpage/article.

How the code works

It’s worth looking at the two parts of the solution that do most of the heavy lifting :

1. Querying Twitter

Querying Twitter is very straight forward using the twit client and requires just one line of code:

T.get('search/tweets', {
	q: process.argv[2],
	count: process.argv[3]
}, function(err, data, response) {
  data.statuses.forEach(function(s) {
...

The above line of code uses the supplied command line arguments to query Twitter, it then passes the returned results one by one to the function that will feed the body of the Tweet and the embedded URL (if any) to the AYLIEN API endpoints for analysis.

2. Analyzing the Tweets that are returned.

The function below, takes the following arguments, the AYLIEN endpoint to call (Sentiment, Hashtags, Entities etc.) the parameters which the endpoint should work on (i.e. we indicate whether we are passing a piece of text or a URL for analysis and we also pass the actual text or URL) and a callback function to call when the analysis is complete.

function call_api(endpoint, parameters, callback) {
  var postData = querystring.stringify(parameters);
  var request = https.request({
    host: 'api.aylien.com',
    path: '/api/v1/' + endpoint,
    headers: {
      'Accept':                             'application/json',
      'Content-Type':                       'application/x-www-form-urlencoded',
      'Content-Length':                     postData.length,
      'X-AYLIEN-TextAPI-Application-ID':    APPLICATION_ID,
      'X-AYLIEN-TextAPI-Application-Key':   APPLICATION_KEY,
    }
  }, function(response) {
    var data = "";
    response.on('data', function(chunk) {
      data += chunk;
    });
    response.on('end', function() {
      callback(JSON.parse(data));
    });
  });
  request.write(postData);
  request.end();
}

The examples we have used, analyze the Tweets for Sentiment and Hashtag Suggestions. It’s up to you what endpoints you wish to use. Maybe you want to extract entities or concepts from the Tweets as well. A full list of our endpoints can be found in our documentation.

The Code Snippet

var Twit = require('./node_modules/twit')  //Twitter API client library
var https = require('https'), querystring = require('querystring');

//AYLIEN API Credentials
const APPLICATION_KEY = YOUR_APPLICATION_KEY,
      APPLICATION_ID = YOUR_APPLICATION_ID;


//Twitter API Credentials
var T = new Twit({
    consumer_key:         YOUR_TWITTER_CONSUMER_KEY
  , consumer_secret:      YOUR_TWITTER_CONSUMER_SECRET
  , access_token:         YOUR_TWITTER_ACCESS_TOKEN
  , access_token_secret:  YOUR_TWITTER_ACCESS_TOKEN_SECRET
})


var analysisResults = {};
var parameters;
var i =  process.argv[3] * 2; //Counter to track when Asynchronous API call have completed

console.log("Processing your request. Please wait...")
console.log("n");

T.get('search/tweets', { q: process.argv[2], count: process.argv[3] }, function(err, data, response) {
  data.statuses.forEach(function(s) {
    var returnedUrls = s.entities.urls;
    analysisResults[s.id] = {};
    analysisResults[s.id].text = s.text;
  	parameters = {'text': s.text};
    callAylienAPIs(parameters,outputResults);

    function callAylienAPIs(parameters, callback) {

      call_api('sentiment', parameters, function(result) {
        var a = {};
        a.endpoint = 'sentiment';
        a.polarity = result.polarity;
        a.polarity_confidence = result.polarity_confidence;
        analysisResults[s.id].sentiment = a;
        i--;
        if (i == 0)  {
          callback();
        }
      })

      if (returnedUrls.length > 0 ) {
          var url_paramaters = {'url' : returnedUrls[0].expanded_url };
          call_api('hashtags', url_paramaters, function(result) {
            var a = {};
            a.endpoint = 'hashtags';
            a.hashtags = result.hashtags;
            analysisResults[s.id].hashtags = a;
            i--;
            if (i == 0)  {
              callback();
            }
          })
        } else {
            var a = {};
            a.endpoint = 'hashtags';
            a.hashtags = 'No Hastags available as no Url specified in the Tweet';
            analysisResults[s.id].hashtags = a;
            i--;
            if (i == 0)  {
              callback();
            }
        }
    }
  });

})


function outputResults() {

    for (var key in analysisResults) {
    console.log("Tweet Text           : ", analysisResults[key].text);
    console.log("Sentiment Polarity   : ", analysisResults[key].sentiment.polarity);
    console.log("Polarity Confidence  : ", analysisResults[key].sentiment.polarity_confidence);
    console.log("Hashtags : ", analysisResults[key].hashtags.hashtags);
    console.log("n");
    }
}




function call_api(endpoint, parameters, callback) {
  var postData = querystring.stringify(parameters);
  var request = https.request({
    host: 'api.aylien.com',
    path: '/api/v1/' + endpoint,
    headers: {
      'Accept':                             'application/json',
      'Content-Type':                       'application/x-www-form-urlencoded',
      'Content-Length':                     postData.length,
      'X-AYLIEN-TextAPI-Application-ID':    APPLICATION_ID,
      'X-AYLIEN-TextAPI-Application-Key':   APPLICATION_KEY,
    }
  }, function(response) {
    var data = "";
    response.on('data', function(chunk) {
      data += chunk;
    });
    response.on('end', function() {
      callback(JSON.parse(data));
    });
  });
  request.write(postData);
  request.end();
}

Author


Avatar

Mike Waldron

Head of Marketing & Sales @ AYLIEN A legal convert with a masters degree from Smurfit Business School, Mike runs our Sales and Marketing at AYLIEN. Mike gathered his Sales and Marketing experience with technology companies in Sydney and Dublin before getting the startup itch and joining the team at AYLIEN. Twitter: @MikeWallly