Good Contents Are Everywhere, But Here, We Deliver The Best of The Best.Please Hold on!
Your address will show here +12 34 56 78

Welcome to the seventh in a series of blog posts in which we use the News API to look into the previous month’s news content. The News API collected and indexed over 2.5 million stories published last month, and in this blog we’re going to use its analytic capabilities to discover trends in what the media wrote about.

We’ve picked two of the biggest stories from last month, and using just three of the News API’s endpoints (Stories, Trends, and Time Series) we’re going to cover the two following topics:

  1. The conflict brewing with a nuclear-armed North Korea and the US
  2. The ‘fight of the century’ between Conor McGregor and Floyd Mayweather

In covering both of these topics we uncovered some interesting insights. First, apparently we’re much more interested in Donald Trump’s musings on nuclear war than the threat of nuclear war itself. Also, although the McGregor fight lived up to the hype, Conor failed to capitalize on the record-breaking press coverage to launch his ‘Notorious Whiskey’.

1. North Korea

Last month, North Korea detonated a Hydrogen bomb, which was over seven times larger than any of their previous tests. This created an increasing worry that conflict with a nuclear-armed nation is now likely. But using the News API, we can see that in the English-speaking world, even with such a threat looming, we still just can’t get enough of Donald Trump.

Take a look below at the daily volume of stories with ‘North Korea’ in the title last month, which we gathered with the News API’s Time Series endpoint. The graph below shows the volume of stories with the term ‘North Korea’ in the title across every day in August. You can see that the English-speaking media were much more interested with Trump’s ‘fire and fury’ comment at the start of August than they were with North Korea actually detonating a Hydrogen bomb at the start of September.



We guessed that this is largely due to publishers trying to keep up with the public’s insatiable appetite for any Donald Trump-related news. Using the News API, we can put this idea to the test, by analyzing what content about North Korea people shared the most over August.

We used the Stories endpoint of the News API to look at the stories that contained ‘Korea,’ in the title that had the highest engagement rates across social networks to understand the type of content people are most likely to recommend in their social circles which gives a strong indication into readers’ opinions and interests. Take a look below at the most-shared stories across Facebook and Reddit. You can see that the popular content varies across the different networks.

Facebook:

Untitled design (12)

 

  1. Trump to North Korea: U.S. Ready to Respond With ‘Fire and Fury,’ The Washington Post. 118,312 shares.
  2. China warns North Korea: You’re on your own if you go after the U.S.’ The Washington Post. 94,818 shares.
  3. Trump threatens ‘fury’ against N Korea,’ BBC. 69,098 shares.

Reddit:

Untitled design (10)

 

  1. Japanese government warns North Korea missile headed toward northern Japan,’  CNBC. 119,075 upvotes.
  2. North Korea shaken by strong tremors in likely nuclear test,’ CNBC. 61,088 upvotes.
  3. Japan, US look to cut off North Korea’s oil supply,’ Nikkei Asian Review. 59,725 upvotes.

Comparing coverage across these three social networks, you can see that Trump features heavily on the most popular content about Korea on Facebook, while the most-upvoted content on Reddit tended to be breaking news with a more neutral tone. This is similar to the patterns we observed with the News API in a previous media review blog, which showed that Reddit was much more focused on breaking stories on popular topics than Facebook.

So now that we know the media focused its attention on Donald Trump, we can ask ourselves, what were all of these stories about? Were these stories talking the President down, like he always claims? Or were they positive? Using the sentiment feature of the News API’s Trends endpoint, we can dive into the stories that had both ‘Trump’ and ‘Korea’ in the title, and see what sentiment is expressed in the body of the text.

From the results below, you can see that over 50% of these articles contained negative sentiment, whereas a little over 30% had a positive tone. For all of the President’s – shall we say, questionable – claims, he’s right about one thing: popular content about how he responds to issues is overwhelmingly negative.


The Superfight – how big was it?

We’re based in Ireland, so having Conor McGregor of all people taking part in the ‘fight of the century’ last month meant that we’ve heard about pretty much nothing else. We can use the News API to put some metrics on all of the hype, and see how the coverage compared to the coverage of other sporting events. Using the Time Series endpoint, we analyzed the impact of the fight on the volume of stories last month. Since it analyzes the content of every news story it gathers, the News API can show us how the volume of stories about every subject fluctuates over time.

Take a look at how the volume of stories about boxing skyrocketed in the build up to and on the weekend of the fight:



You can see that on the day of fight itself, the volume of stories that the News API classified as being about boxing increased, almost by a factor of 10.

To figure out just how big this hype was in the boxing world, we compared the volume of stories published about boxing in the time period surrounding the ‘fight of the century’ and another boxing match which at the time received a lot of hype, the WBA/IBF world heavyweight title bout last April between Anthony Joshua and Wladimir Klitschko. In order to do this, we analyzed the story volume from the two weeks before and after each fight and plotted them side by side. This allows us to easily compare the media coverage on the day of the fight as well as its build-up and aftermath. Take a look below at the results below:



You can see that the McGregor-Mayweather fight totally eclipses the Joshua-Klitschko heavyweight title fight. But it’s important to give context to the data on this hype by comparing it with data from other sports.

It’s becoming almost a point of reference on these News API media review blogs to compare any trending stories to stories in the World Soccer category. This is because the daily volume of soccer stories tends to be consistently the largest of all categories, so it’s a nice baseline to use to compare story volumes. As you can see below, the hype surrounding the ‘fight of the century’ even prompted more boxing stories than soccer stories, which is quite a feat. Notice how only four days after the fight, when boxing was back to its normal level and soccer stories were increasing due to European transfer deadline day looming, there were 2,876 stories about soccer compared with 191 stories about boxing.

You might remember Conor McGregor launched his ‘Notorious Whiskey’ in the press conference following the fight. This was the perfect time for McGregor to launch announce a new product – right at the pinnacle of the media coverage. If you’re wondering how well he leveraged this phenomenal level of publicity for his new distilling career, we used the News API to look into that too. Take a look below at volume of stories that mentioned the new whiskey brand. It looks like mentions of ‘Notorious Whiskey’ have disappeared totally since the weekend of the fight, leaving us with this odd-looking bar chart. But we doubt that will bother Conor at the moment, considering the $100m payday!

That covers our quick look into the News API’s data on two of last month’s stories. The News API gathers over 100,000 stories per day, and indexes them in near-real time. This gives you a stream of enriched news data that you can query. So try out the demo or click on the link below for a free trial to use our APIs for two weeks.






News API - Sign up





0

At AYLIEN, we provide the building blocks for our customers to create Natural Language Processing-powered solutions. To give you an idea of what some of these solutions look like, we occasionally put together use cases (check out how customers like Complex Media and Streem use our APIs).

For this blog, we’re going to show how 1043 Labs, a US-based software consultancy firm, used our Text Analysis API as part of an innovative platform they built for a client that recently closed a $5 million funding round to expand their operations.


1043

 

1043 Labs is a custom software consulting firm that helps entrepreneurs get their ideas off the ground. They are a team made up experts from a wide variety of technology fields who bring their technical expertise to the table to help their customers build products, services, and solutions. So when Chris Kraft founded Share Rocket with a vision to create a ratings system for digital media, he hired 1043 Labs to help build the solution he had envisioned.

The Challenge

In the US, 2016 saw digital ad spending surpass that of television ads, with the global annual digital ad spend hitting $72.5 billion in 2016, a figure that’s set to continue to grow into the future. But to get a slice of this digital ad spend, publishers and media organizations across the board need to understand how their content performs so that they can make data-driven decisions on their strategies.

 

Share-Rocket

 

This is where Share Rocket can help. Using a collection of digital tools, Share Rocket’s customers like NBC, Fox, and Hearst can measure how successful their content is online. These tools make it easy to monitor social reach and understand how successful their content is online. Information and data obtained from Share Rocket help users to assess what type of content they need to produce more of. In turn, this allows them to make data-driven decisions on what content to produce and how to promote it, maximizing potential ad revenue.

 

Share Rocket

 

How does AYLIEN fit in?

To understand what content performs best, Share Rocket needs to understand what every piece of content is about. This allows their users to ask questions about the subject matter of their text content – what isn’t working, and what they should produce more of. Moreover, Share Rocket needs to understand this at scale. Their clients produce thousands of pieces of content per hour which means manually tagging all of this content would be almost impossible due to the scale alone.

 

AYLIEN 1043 Case Study

 

So when Share Rocket hired 1043 Labs to help build their platform, the consultants at 1043 Labs assessed a number of Natural Language Processing APIs with a clear idea of what exactly they were looking for and prioritized the following requirements:

  • Accuracy
  • Speed
  • Customer support  
  • Time to value

Following some in-depth testing and discussions with Mike and some of the engineering team, 1043 Labs chose the AYLIEN Text Analysis API.

“AYLIEN saved us about 3 person months of development. This made our client happier because we delivered the project earlier than planned and under budget.”

Mike Ostman, Founding Partner, 1043 Labs

 

We’ve heard Mike’s sentiment echoed in calls with our customers quite a bit. People respond really well to how easy our tools are to integrate into whatever you’re building – signing up takes a couple of minutes, and you’ll be making your first calls to the API with a couple of lines of code. After deciding to go with AYLIEN, the team at 1043 Labs had their entire solution built in a couple of weeks.

How are Share Rocket using the AYLIEN Text API?

Share Rocket’s users need accurate metrics on the performance of their content online, so to provide the full picture, Share Rocket need to understand what every piece of content is about. This allows their users to ask questions like ‘how do people like our sports coverage?’ Or ‘is our coverage of weather doing better than our competitor’s?’

 

Monetize - Share Rocket

But the problem here is that text content is unstructured, meaning it’s particularly difficult for computers to analyze and understand. This is where the Natural Language Processing comes in. Every time a new piece of content is published by their users, Share Rocket use the Classification endpoint of our Text Analysis API to understand what the content is about. Our Classification feature allows users to categorize content based on two industry standard taxonomies: IAB-QAG and IPTC. The ability to classify content automatically means Share Rocket can utilize assigned categories as tags in their proprietary SHARE and SEI tools. These tags can then be used to track content performance across subject categories.

Share Rocket now analyze upwards of 150,000 pieces of content every day for their users with our API. This is a testament not only to the fact that they’ve built a service people need, but also to the fact that our Text Analysis API is a robust tool that scales as your demand increases.

Getting started with our APIs is simple. Open your account by clicking on the CTA below. Once you’ve created your account you can start calling the API within minutes, with a few lines of code.




Text Analysis API - Sign up

0

Breakthroughs in NLP research are creating huge value for people every day, supercharging technologies from search engines to chatbots. The work that makes these breakthroughs possible is done in two silos – academia and industry. Researchers in both of these silos produce work that advances the field, and frequently collaborate to generate innovative research.

Contributing to this research is why we have such a heavy R&D focus at AYLIEN, with six full-time research scientists out of a total team of 16. The research team naturally has strong ties with academia – some are completing PhDs with the work they are carrying out here, while others already hold one. Academia is also represented on our advisory board and in the great people who have become our mentors.

To further deepen these ties with academia, we’re delighted to announce our first Industry Fellowship in association with Science Foundation Ireland, with Dr. Ian Wood of NUIG. Ian will be based in our Dublin office for one year starting in September. SFI’s goal with this fellowship is to allow industry and academia to cross-pollinate by exchanging ideas and collaborating on research. This placement will allow us to contribute to and learn from the fantastic work that Insight Centre in NUIG are doing, and we’re really excited to open up some new research windows where our team’s and Ian’s interests overlap.

Screenshot (352)

Ian is a postdoctoral researcher at the Insight Centre for Data Analytics, with an incredibly interesting background – a mixture of pure Mathematics, Psychology, and Deep Learning. His research is focused on how the emotions of entire communities change over time, which he researches by creating language models that detect the emotions people express on social media. For his PhD, he analyzed Tweets produced by pro-anorexia communities over three years and tracked their emotions, and showed that an individual’s actions are much more driven by their surrounding community than is generally accepted. Continuing this research, Ian now specializes in finding new ways to build Machine Learning and Deep Learning models to analyze emotions in online communities.

Ian’s placement is mutually beneficial on two levels. First, Ian’s experience in building language models for emotion analysis is obviously beneficial to us, and we can offer Ian a cutting edge research infrastructure and the opportunity to learn from our team in turn. But we’re also really excited at the possibility of opening up new research areas based on common interests, for example by building on existing research between Ian and our PhD student, Sebastian. Ian’s research into reducing dimensionality in data sets crosses over with Sebastian’s work into Domain Adaptation in a really interesting way, and we’re excited that this could open up a new research area for us to work on.

Outside of AYLIEN, Ian also speaks four languages, he was a professional musician (but that was in a previous life, he tells us), and he’s also sailed across the Atlantic in a small boat, so he’ll hopefully have some input into the next AYLIEN team-building exercises…

Welcome to the team, Ian!

If you want to find out more about the Fellowship, check out the LinkedIn group, and if your research interests overlap with ours in any way, drop us a line at jobs@aylien.com – we love hearing from other researchers!

0

In 2017, video content is becoming ever more central to how people consume media. According to research by HighQ, this year around 30% of smartphone users will watch video content on their device at least once a day. In addition to this, people will spend on average an extra two minutes browsing sites that feature video content compared with sites that do not. For this reason, video content is an important component of driving up revenues for online news publishers, since keeping your audience on your site allows you to sell more ads.

But even though we can find great market research on consumer behavior around video content, we couldn’t find an answer to the following question — what type of video content is the news industry publishing to capitalize on this? For example, how much video content is actually being published? Are some publishers dominating video content? And are some subjects being supplemented with videos more regularly than others? Knowing this would allow us to understand what areas of the online news industry are set to flourish in the coming years with the growing emphasis on video.

We decided to use the News API to look into this question. Last month, our API crawled, analyzed, and indexed 1,344,947 stories as they were published. One of the metadata points that it analyzed was how many images and videos were embedded on the page. So for this blog, we’ll analyze the 1.3 million stories our News API gathered in July to find answers to the following questions:

  1. How many of the stories published last month featured video content?
  2. What were the stories with video content about?
  3. Which news organizations published the most video content?

1. How many stories published last month contained video content?

To get an idea of how far the video medium has spread into the online news industry, we need to find how much video content was used by news publishers last month. To do this, we used the News API’s Time Series endpoint to sort the stories published in July according to how many videos they contained. We then visualized the results to show how many stories contained no videos, how many contained one video, and how many contained more than one. Take a look below at what we found:

As you can see, 96% of stories published last month did not contain any video content, whereas just under 4% contained one video or more. We found this interesting — while HighQ found that almost 30% of smartphone users will watch video content online at least once per day, we can see here that barely 3.5% of news content published last month contained a video. This isn’t really optimal for an industry that relies on clicks for ad revenue.

But let’s focus on the news stories that contained video content. If we knew what these stories were about, we would have a good idea about what areas of online news are likely to fare well, since these areas likely account for a large proportion of ad revenue, and are therefore likely to grow. To look into this, we decided to try to understand what the stories containing video content were about.

2. What were the stories containing video about?

Knowing that only around one out of every thirty stories contained video content last month is interesting, but it begs the question of what these stories were about. To answer this question, we used the Trends endpoint to analyze the 43,134 stories that contained one video and see what subjects each one was about.

One of the pieces of information our News API extracts is topics that are discussed in the story, and which categories the story fits into, based on two taxonomies. For this visualization, we’ll use the advertising industry’s IAB-QAG taxonomy. Take a look below at which categories contained the most video content:

You can see that the Entertainment category had the most stories with video content accompanying them. This isn’t surprising to us at first, as we have all seen articles about celebrities with annoying videos that play automatically. But if you remember last month’s media roundup, you’ll remember that the Sports and Law, Government, and Politics categories produced by far the highest volumes of content (the Sports category alone published over double the content of the Entertainment category). This means that not only are there more videos about entertainment, but also that stories about entertainment are much more likely to contain a video than stories about politics.

So now we know which subject categories video content appeared in the most. But with the News API, we can go one step further and see exactly what people were talking about in the stories that contained a video. To do this, we used the Trends endpoint again to extract the entities mentioned the titles of these stories. Take a look at the chart below to see what people were talking about:

Here you can see exactly what the stories containing videos were about. The single biggest subject that was accompanied by a video was Love Island, a reality TV show. But you can also see that large soccer clubs are well represented on the chart. If you think back to last month’s roundup again, you’ll remember the huge reach and popularity of the top soccer clubs, even during their off-season. The chart above shows that these large soccer clubs are also being covered more with video content than other entities, with publishers obviously trying to leverage this reach to attract people to the stories they publish.

With large soccer clubs dominating both regular news content and video news content, and with ad revenues for video content being so valuable, these soccer clubs look like they have a bright future in terms of media content. Since the clubs benefit financially from media coverage through things like player image rights and viewership of games, large transfer fees like the $263 million PSG are going to pay for Neymar don’t look so crazy.

3. Who were the biggest publishers of video content?

As we mentioned in the introduction, we want to find out which publishers are making the quickest transition to video-based content, as this has a knock-on effect on site viewership, and therefore ad revenues. Knowing which players are leading industry trends like this is a good indicator of which ones are going to survive in an industry that is under financial pressure while transitioning to digital.

With that in mind, we used the Trends endpoint to find out which publishers were leading the way in video content. You can see pretty clearly from the graph below that the Daily Mail dominates last month’s video content. To see the rest of the publishers more clearly, you can select the Daily Mail bubble below and click “exclude”.

The Daily Mail obviously dominate the chart here, which isn’t too surprising when you consider that they feature video as a central part of the content on their site. They produce a huge amount of stories every month, and feature video even when it wasn’t completely related to the story it appeared with. Although the discontinuity can seem odd, even a loosely-related video can increase click through rate and revenues.

As you can see, many traditional news publishers are lagging behind in terms of the amount of video they’re publishing, with The Guardian, Forbes, ABC, and The Daily Mail among the few recognizable print and television giants on the graph. Instead, the field is largely made up of publishers like The Elite Daily, Uproxx, and Heavy, digital native organizations who are publishing more online video content than most traditional publishers.

Well, that concludes our brief analysis of last month’s video content in news stories. If you’re an AYLIEN subscriber, we’d like to remind you that the two endpoints we used in this post (Trends and Time Series) do not return stories, so you can hit them as much as you like and they won’t contribute towards your monthly 10,000 stories. So dig in!

If you’re not a subscriber, you can try the News API free of charge for two weeks by clicking on the image below (free means free, there’s no card required or obligation to buy).





News API - Sign up




0




Every day, we generate huge amounts of text online, creating vast quantities of data about what is happening in the world and what people think. All of this text data is an invaluable resource that can be mined in order to generate meaningful business insights for analysts and organizations. However, analyzing all of this content isn’t easy, since converting text produced by people into structured information to analyze with a machine is a complex task. In recent years though, Natural Language Processing and Text Mining has become a lot more accessible for data scientists, analysts, and developers alike.

There is a massive amount of resources, code libraries, services, and APIs out there which can all help you embark on your first NLP project. For this how-to post, we thought we’d put together a three-step, end-to-end guide to your first introductory NLP project. We’ll start from scratch by showing you how to build a corpus of language data and how to analyze this text, and then we’ll finish by visualizing the results.

We’ve split this post into 3 steps. Each of these steps will do two things: show a core task that will get you familiar with NLP basics, and also introduce you to some common APIs and code libraries for each of the tasks. The tasks we’ve selected are:

  1. Building a corpus — using Tweepy to gather sample text data from Twitter’s API.
  2. Analyzing text — analyzing the sentiment of a piece of text with our own SDK.
  3. Visualizing results — how to use Pandas and matplotlib to see the results of your work.

Please note: This guide is aimed at developers who are new to NLP and anyone with a basic knowledge of how to run a script in Python. If you don’t want to write code, take a look at the blog posts we’ve put together on how to use our RapidMiner extension or our Google Sheets Add-on to analyze text.

 
pythonlogo

Step 1. Build a Corpus

You can build your corpus from anywhere — maybe you have a large collection of emails you want to analyze, a collection of customer feedback in NPS surveys that you want to dive into, or maybe you want to focus on the voice of your customers online. There are lots of options open to you, but for the purpose of this post we’re going to use Twitter as our focus for building a corpus. Twitter is a very useful source of textual content: it’s easily accessible, it’s public, and it offers an insight into a huge volume of text that contains public opinion.

Accessing the Twitter Search API using Python is pretty easy. There are lots of libraries available, but our favourite option is Tweepy. In this step, we’re going to use Tweepy to ask the Twitter API for 500 of the most recent Tweets that contain our search term, and then we’ll write the Tweets to a text file, with each Tweet on its own line. This will make it easy for us to analyze each Tweet separately in the next step.

You can install Tweepy using pip:

pip install tweepy

Once completed, open a Python shell to double-check that it’s been installed correctly:

>>> import tweepy

First, we need to get permission from Twitter to gather Tweets from the Search API, so you need to sign up as a developer to get your consumer keys and access tokens, which should take you three or four minutes. Next, you need to build your search query by adding your search term to the q = ‘’ field. You will also need to add some further parameters like the language, the amount of results you want returned, and the time period to search in. You can get very specific about what you want to search for on Twitter; to make a more complicated query, take a look at the list of operators you can use the API to search with in the Search API introduction.

Fill your credentials and your query into this script:


## import the libraries
import tweepy, codecs

## fill in your Twitter credentials 
consumer_key = ‘your consumer key here’
consumer_secret = ‘your consumer secret key here’
access_token = ‘your access token here’
access_token_secret = ‘your access token secret here’

## let Tweepy set up an instance of the REST API
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

## fill in your search query and store your results in a variable
results = api.search(q = "your search term here", lang = "en", result_type = "recent", count = 1000)

## use the codecs library to write the text of the Tweets to a .txt file
file = codecs.open("your text file name here.txt", "w", "utf-8")
for result in results:
	file.write(result.text)
	file.write("\n")
file.close()

You can see in the script that we are writing result.text to a .txt file and not simply the result, which is what the API is returning to us. APIs that return language data from social media or online journalism sites usually return lots of metadata along with your results. To do this, they format their output in JSON, which is easy for machines to read.

For example, in the script above, every “result” is its own JSON object, with “text” being just one field — the one that contains the Tweet text. Other fields in the JSON file contain metadata like the location or timestamp of the Tweet, which you can extract for a more detailed analysis.

To access the rest of the metadata, we’d need to write to a JSON file, but for this project we’re just going to analyze the text of people’s Tweets. So in this case, a .txt file is fine, and our script will just forget the rest of the metadata once it finishes. If you want to take a look at the full JSON results, print everything the API returns to you instead:

This is also why we used codecs module, to avoid any formatting issues when the script reads the JSON results and writes utf-8 text.

Step 2. Analyze Sentiment

So once we’ve collected the text of the Tweets that you want to analyze, we can use more advanced NLP tools to start extracting information from it. Sentiment analysis is a great example of this, since it tells us whether people were expressing positive, negative, or neutral sentiment in the text that we have.

For sentiment analysis, we’re going to use our own AYLIEN Text API. Just like with the Twitter Search API, you’ll need to sign up for the free plan to grab your API key (don’t worry — free means free permanently. There’s no credit card required, and we don’t harass you with promotional stuff!). This plan gives you 1,000 calls to the API per month free of charge.

Again, you can install using pip:


pip install aylien-apiclient

Then make sure the SDK has installed correctly from your Python shell:


>>>from aylienapiclient import textapi

Once you’ve got your App key and Application ID, insert them into the code below to get started with your first call to the API from the Python shell (we also have extensive documentation in 7 popular languages). Our API lets you make your first call to the API with just four lines of code:


>>>from aylienapiclient import textapi
>>>client = (‘Your_app_ID’, ‘Your_application_key’)
>>>sentiment = client.Sentiment({'text': 'enter some of your own text here'})
>>>print(sentiment)

This will return JSON results to you with metadata, just like our results from the Twitter API.

So now we need to analyze our corpus from step 1. To do this, we need to analyze every Tweet separately. The script below uses the io module to open up a new .csv file and write the column headers “Tweet” and “Sentiment”, and then it opens and reads the .txt file containing our Tweets. Then, for each Tweet in the .txt file it sends the text to the AYLIEN API, extracts the sentiment prediction from the JSON that the AYLIEN API returns, and writes this to the .csv file beside the Tweet itself.

This will give us a .csv file with two columns — the text of a Tweet and the sentiment of the Tweet, as predicted by the AYLIEN API. We can look through this file to verify the results, and also visualize our results to see some metrics on how people felt about whatever our search query was.


from aylienapiclient import textapi
import csv, io

## Initialize a new client of AYLIEN Text API
client = textapi.Client("your_app_ID", "your_app_key")

with io.open('Trump_Tweets.csv', 'w', encoding='utf8', newline='') as csvfile:
	csv_writer = csv.writer(csvfile)
	csv_writer("Tweet", " Sentiment")
	with io.open("Trump.txt", 'r', encoding='utf8') as f:
	    for tweet in f.readlines():
	    	## Remove extra spaces or newlines around the text
	    	tweet = tweet.strip()

	    	## Reject tweets which are empty so you don’t waste your API credits
	    	if len(tweet) == 0:
	    		print('skipped')
	    		continue
	    	
	    	print(tweet)

	    	## Make call to AYLIEN Text API
	    	sentiment = client.Sentiment({'text': tweet})

	    	## Write the sentiment result into csv file
	    	csv_writer.writerow([sentiment['text'], sentiment['polarity']])

You might notice on the final line of the script that when the script goes to write the Tweet text to the file, we’re actually writing the Tweet as it is returned by the AYLIEN API, rather than the Tweet from the .txt file. They are both identical pieces of text, but we’ve chosen to write the text from the API just to make sure we’re reading the exact text that the API analyzed. This is just to make it clearer if we’ve made an error somehow.

Step 3. Visualize your Results

So far we’ve used an API to gather text from Twitter, and used our Text Analysis API to analyze whether people were speaking positively or negatively in their Tweet. At this point, you have a couple of options with what you do with the results. You can feed this structured information about sentiment into whatever solution you’re building, which could be anything from a simple social listening app or a even an automated report on the public reaction to a campaign. You could also use the data to build informative visualizations, which is what we’ll do in this final step.

For this step, we’re going to use matplotlib to visualize our data and Pandas to read the .csv file, two Python libraries that are easy to get up and running. You’ll be able to create a visualization from the command line or save it as a .png file.

Install both using pip:


pip install matplotlib
pip install pandas

The script below opens up our .csv file, and then uses Pandas to read the column titled “Sentiment”. It uses Counter to count how many times each sentiment appears, and then matplotlib plots Counter’s results to a color-coded pie chart (you’ll need to enter your search query to the “yourtext” variable for presentation reasons).


## import the libraries
import matplotlib.pyplot as plt 
import pandas as pd
from collections import Counter
import csv 

## open up your csv file with the sentiment results
with open('your_csv_file_from_step_3', 'r', encoding = 'utf8') as csvfile:
	## use Pandas to read the “Sentiment” column,
df = pd.read_csv(csvfile)
	sent = df["Sentiment"]

## use Counter to count how many times each sentiment appears
## and save each as a variable
	counter = Counter(sent)
	positive = counter['positive']
	negative = counter['negative']
	neutral = counter['neutral']

## declare the variables for the pie chart, using the Counter variables for “sizes”
labels = 'Positive', 'Negative', 'Neutral'
sizes = [positive, negative, neutral]
colors = ['green', 'red', 'grey']
yourtext = "Your Search Query from Step 2"

## use matplotlib to plot the chart
plt.pie(sizes, labels = labels, colors = colors, shadow = True, startangle = 90)
plt.title("Sentiment of 200 Tweets about "+yourtext)
plt.show()

If you want to save your chart to a .png file instead of just showing it, replace plt.show on the last line with savefig(‘your chart name.png’). Below is the visualization we ended up with (we searched “Trump” in step 1).

Screenshot (261)

If you run into any issues with these scripts, big or small, please leave a comment below and we’ll look into it. We always try to anticipate any problems our own users might run into, so be sure to let us know!

That concludes our introductory Text Mining project with Python. We hope it gets you up and running with the libraries and APIs, and that it gives you some ideas about subjects that would interest you. With the world producing content on such a large scale, the only obstacle holding you back from an interesting project is your own imagination!

Happy coding!





Text Analysis API - Sign up




1

Chatbots are a hot topic in tech at the moment. They’re at the center of a shift in how we communicate, so much so that they are central to the strategy and direction of major tech companies like Microsoft and Facebook. According to Satya Nadella, CEO of Microsoft, “Chatbots are the new apps”.

So why exactly have chatbots become so popular?

Their rise in popularity is partly connected to the resurgence of AI and its applications in industry, but it’s also down to our insatiable appetite for on-demand service and our shift to messaging apps over email and phone. A recent study found that 44% of US consumers would prefer to use chatbots over humans for customer relations, and 61% of those surveyed said they interact with a chatbot at least once a month. This is because they suit today’s consumers’ needs – they can respond to customer queries instantly, day or night. 

Large brands and tech companies have recognised this shift in customer needs and now rely on messenger and intelligent assistants to provide a better experience for their customers. This is especially true since Facebook opened up its Messenger platform to third-party bots last year.

So while the adoption of intelligent assistants and chatbots is growing at a colossal rate, contrary to popular belief and media hype, they’re actually nothing new. We’ve had them for over fifty years in the Natural Language Processing community and they’re a great example of the core mission of NLP  – programming computers to understand how humans communicate.

In this blog, we’re going to show three different chatbots and let you interact with each bot so you can see how they have advanced. We’ll give some slightly technical explanations of how each chatbot works so you can see how NLP works under the hood.

The Chatbots

The three chatbots we’ve gathered on this page are:

  1. ELIZA – a chatbot from 1966 that was the first well-known chatbot in the NLP community
  2. ALICE – a chatbot from the late 1990s that inspired the movie Her
  3. Neuralconvo – a Deep Learning chatbot from 2016 that learned to speak from movie scripts

Chatbots Blog

We should mention here that these three bots are all “chit-chat” bots, as opposed to “task-oriented” bots. Whereas task-oriented bots are built for a specific use like checking if an item is in stock or ordering a pizza, a chit-chat bot has no function other than imitating a real person for you to chat with. By seeing how chit-chat bots have advanced, you’re going to see how the NLP community has used different methods to replicate human communication.

ELIZA – A psychotherapy bot

The first version of ELIZA was finished in 1966 by Joseph Weizenbaum, a brilliant, eccentric MIT professor considered one of the fathers of AI (and who is the subject of a great documentary). ELIZA emulates a psychotherapist, one that Weizenbaum’s colleagues trusted enough to divulge highly personal information, even after they knew it was a computer program. Weizenbaum was so shocked at how his colleagues thought ELIZA could help them, even after they knew it was a computer program, that he spent the rest of his life advocating for social responsibility in AI.

ELIZA

But ELIZA only emulates a psychotherapist because it uses clever ways to return your text as a question, just like a real psychotherapist would. This clever tactic means ELIZA can respond to a question that it doesn’t understand with a relatively simple process of rephrasing the input as a question, so the user is kept in conversation.

Just like any algorithm, chatbots work from rules that tell it how to take an input and produce an output. In the case of chatbots, the input is text you supply to it, and the output is text it returns back to you as a response. Looking at the responses you get from ELIZA, you’ll see two rough categories of rules:

  • on a syntactic level, it transfers personal pronouns (“my” to “your,” and vice versa).
  • to imitate semantic understanding (ie that it understands the meaning of what you are typing), it has been programmed to recognize certain keywords and returns phrases that have been marked as suitable returns to the input. For instance, if you input “I want to ___” it will return “What would it mean to you if you ___”.

Try and figure out some of ELIZA’s limits for yourself by asking it questions and trying to figure out why it’s returning each of its responses. Remember: it’s from the 1960s, when color televisions were the height of consumer technology.

Eliza



This is a pure Natural Language Processing approach to building a chatbot: the bot understands human language by the rules mentioned above, which are basically grammar rules programmed into a computer. This achieves impressive results, but if you wanted to make ELIZA more human-like by pure NLP methods you would have to add more and more grammatical rules, and because grammar is complicated and contradictory, you would quickly end up with a sort of “rule spaghetti,” which wouldn’t work. This approach is in contrast with Machine Learning approaches to chatbots (and natural language in general), where an algorithm will try to guess the correct response based on observations it has made on other conversations. You can see this in action in the final chatbot, Neuralconvo. But first, ALICE.

ALICE – The star of the movie Her

Fast forward from the 1960s to the late 1990s and you meet ALICE, the first well-known chatbot that people could interact with online, and one that developed something of a cult reputation. Director Spike Jonze said that chatting with ALICE in the early 2000s first put the idea for his 2013 film Her in his mind, a movie where a man falls in love with the AI that powers his operating system.

her

But just like ELIZA, this is a computer program made up of rules that take an input and produce an output. Under the hood, ALICE is an advance on ELIZA in three respects:

  • it is written in a programming language called Artificial Intelligence Markup Language (AIML), similar to XML, which allows it to choose responses on a more abstract level
  • it contains tens of thousands of possible responses
  • it stores previous conversations with users and adds them to its database.

ALICE is an open source bot, one that anyone can download and modify or contribute to. Written originally by Dr. Richard Wallace, over 500 volunteers have contributed to the bot, creating 100,000s of lines of AIML for ALICE to reproduce in conversation.

So ALICE’s improvements on ELIZA allow for more responses that are better tailored to the text you are supplying it with. This allows ALICE to impersonate a person in general, rather than a therapist specifically. The problem here is that the shortcomings are now more obvious – without open ended statements and questions, the lack of a response that matches your input is more obvious. Explore this for yourself below.



So even though ALICE is a more advanced chatbot than ELIZA, the output responses are still written by people, and algorithms choose which output best suits the input. Essentially, people type out the responses and write the algorithms that choose which of these responses will be returned in the hope of mimicking an actual conversation.

Improving the performance and intelligence of chatbots is a popular research area and much of the recent interest in advancing chatbots has been around Deep Learning. Applying Deep Learning to chatbots seems likely to massively improve a chatbot’s ability to interact more like a human. Whereas ELIZA and ALICE reproduce text that was originally written by a person, a Deep Learning bot creates its own text from scratch, based on human speech it has analyzed.

Neuralconvo – A Deep Learning bot

One such bot is Neuralconvo, a modern chatbot created in 2016 by Julien Chaumond and Clément Delangue, co-founders of Huggingface, which was trained using Deep Learning. Deep Learning is a method of training computers to learn patterns in data by using deep neural networks. It is enabling huge breakthroughs in computer science, particularly in AI, and more recently NLP. When applied to chatbots, Deep Learning allows programs to select a response or even to generate entirely new text.

Neuralconvo can come up with its own text because it has “learned” by reading thousands of movie scripts and recognizing patterns in the text. So when Neuralconvo reads a sentence it recognizes patterns in your text, refers back to its training to look for similar patterns, and then generates a new sentence for you that it thinks would follow your sentence if it were in the movie scripts in a conversational manner. It’s basically trying to be cool based on movies it’s seen.

The fundamental difference between ELIZA and Neuralconvo is this: whereas ELIZA was programmed to respond to specific keywords in your input with specific responses, Neuralconvo is making guesses based on probabilities it has observed in movie scripts. So there are no rules telling Neuralconvo to respond to a question a certain way, for example, but the possibilities of its answers are limitless.

Considering Neuralconvo is trained on movie scripts, you’ll see that its responses are suitably dramatic.



The exact model that is working under the hood here is based on the Sequence to Sequence architecture, which was first applied to generate dialogue by Quoc Viet Le and Oriol Vinyals. This architecture consists of two parts: the first one encodes your sentence into a vector, which is basically a code that represents the text. After the entire input text has been encoded this way, the second part then decodes that vector and produces the answer word-by-word by predicting each word that is most likely to come next.

Neuralconvo isn’t going to fool you into thinking that it is a person anytime soon, since it is just a demo of a bot trained on movie scripts. But imagine how effective a bot like this could be when trained using context-specific data, like your own SMS or WhatsApp messages. That’s what’s on the horizon for chatbots, but remember – they will still be algorithms taking your text as input, referring to rules, and returning different text as an output.

Well that sums up our lightning tour of chatbots from the 1960s to today. If you’re interested in blogs about technical topics like training AI to play Flappy Bird or why you should open-source your code, take a look at the Research section of our blog, where our research scientists and engineers post about what interests them.




Text Analysis API - Sign up










0

Extracting insights from millions of articles at once can create a lot of value, since it lets us understand what information thousands of journalists are producing about what’s happening in the world. But extracting accurate insights depends on filtering out noise and finding relevant content. To allow our users access to relevant content, our News API analyzes thousands of news articles in near real-time and categorizes them according to what content is about.

Having content at web-scale arranged into categories provides accurate information about what the media are publishing as the stories emerge. This allows us to do two things, depending on what we want to use the API for: we can either look at a broad picture of what is being covered in the press, or we can carry out a detailed analysis of the coverage about a specific industry, organization, or event.

For this month’s roundup, we decided to do both. First we’re going to take a look at what news categories the media covered the most to see what the content is about in the most written-about categories, and then we’ll pick one category for a more detailed look. First we’ll take a high-level look at sports content, because it’s what the world’s media wrote the most about, and then we’ll dive into stories about finance, to see what insights the News API can produce for us in a business field.

The 100 categories with the highest volume of stories

The range of the subject matter contained in content published every day is staggering, which makes understanding all of this content at scale particularly difficult. However, the ability to classify new content based on well known, industry-standard taxonomies means it can be easily categorized and understood.

Our News API categorizes every article it analyzes according to two taxonomies: Interactive Advertising Bureau’s QAG taxonomy and IPTC’s Newscodes. We chose to use the IAB-QAG taxonomy, which contains just under 400 categories and subcategories, and decided to look into the top 100 categories and subcategories that the media published the most about in June. This left us with just over 1.75 million of the stories that our News API has gathered and analyzed.

Take a look at the most popular ones in the visualization below.

Note: you can interact with all of the visualizations on this blog – click on each data point for more information, and exclude the larger data points if you want to see more detail on the smaller ones.



As you can see, stories about sport accounted for the most stories published in June. It might not surprise people to see that the media publish a lot about sport, but the details you can pick out here are pretty interesting – like the fact that there were more stories about soccer than food, religion, or fashion last month.

The chart below puts the volume of stories about sports into perspective – news outlets published almost 13 times more stories about sports than they did about music.


What people wrote about sports

Knowing that people wrote so much about sport is great, but we still don’t know what people were talking about in all of this content. To find this out, we decided to dive into the stories about sports and see what the content was about – take a look at the chart below showing the most-mentioned sports sub-categories last month.

In this blog we’re only looking into stories that were in the top 100 sub-categories overall, so if your favourite sport isn’t listed below, that means it wasn’t popular enough and you’ll need to query our API for yourself to look into it (sorry, shovel racers).



You can see how soccer dominates the content about sport, even though it’s off-season for every major soccer league. To put this volume in perspective, there were more stories published about soccer than about baseball and basketball combined. Bear in mind, last month saw the MLB Draft and the NBA finals, so it wasn’t exactly a quiet month for either of these sports.

We then analyzed the stories about soccer with the News API’s entities feature to see what people, countries, and organisations people were talking about.



If you check the soccer schedules for June, you’ll see the Confederations Cup is the only major tournament taking place, which is a competition between international teams. However you can see above that the soccer coverage was still dominated by stories about the clubs with the largest fan bases. The most-mentioned clubs above also top the table in a Forbes analysis  f clubs with the greatest social media reach among fans.

Finance

So we’ve just taken a look at what people and organizations dominated the coverage in the news categories that the media published the most in. But even though the sports category is the single most popular one, online content is so wide-ranging that sports barely accounted for 10% of the 1.75 million stories our News API crawled last month.

We thought it would be interesting to show you how to use the API to look into business fields and spot a high-level trend in the news content last month. Using the same analysis that we used on sports stories above, we decided to look at stories about finance. Below is a graph of the most-mentioned entities in stories published in June that fell into the finance category.



You can see that the US and American institutions dominate the coverage of the financial news. This is hardly surprising, considering America’s role as the main financial powerhouse in the world. But what sticks out a little here is that the Yen is the only currency entity mentioned, even though Japan isn’t mentioned as much as other countries.

To find out what kind of coverage the Yen was garnering last month, we analyzed the sentiment of the stories with “Yen” in the title to see how many contained positive, negative, or neutral sentiment.



We can see that there is much more negative coverage here than positive coverage, so we can presume that Japan’s currency had some bad news last month, but that leaves with a big question: why was there so much negative press about the Yen last month?

To find out, we used the keywords feature. Analyzing the keywords in stories returns more detailed information than the entities endpoint we used on the soccer content above, so it is best used when you’re diving into a specific topic rather than getting an overview of some news content, since you’ll get a lot of noise then. It is more detailed because whereas the entities feature returns accurate information about the places, people, and organisations mentioned in stories, the keywords feature will also include the most important nouns and verbs in these stories. This means that we can see a more detailed picture of the things that happened.

Take a look below at the most-mentioned keywords from stories that were talking about the Yen last month.



You can see that the keywords feature returns a different kind of result than entities – words like “year,” and “week,” and “investor,” for example. If we looked at the keywords from all of the news content published in June, it would be hard to get insights because the keywords would be so general. But since we’re diving into a defined topic, we can extract some detailed insights about what actually happened.

Looking at the chart above you can probably guess for yourself what the main stories about the Yen last month involved. We can see from the fact that the most-mentioned terms above that keywords like “data,’ “growth,” “GDP,” and “economy” that Japan has had some negative data about economic growth, which explains the high volume of negative stories about the Yen. You can see below how the value of the Yen started a sustained drop in value after June 15th, the day this economic data was announced, and our News API has tracked the continued negative sentiment.

yen to usd

These are just a couple of examples of steps our users take to automatically extract insights from content on subjects that interest them, whether it is for media monitoring, content aggregation, or any of the thousands of use cases our News API facilitates.

If you can think of any categories you’d like to extract information from using the News API, sign up for a free 14-day trial by clicking on the link below (free means free – you don’t need a credit card and there’s no obligation to purchase).




News API - Sign up




0

Every day, over 100,000 flights carry passengers to and from destinations all around the world, and it’s safe to say air travel brings out a fairly mixed bag of emotions in people. Through social media, customers now have a platform to say exactly what’s on their mind while they are traveling, creating a real-time stream of customer opinion on social networks.

If you follow this blog you’ll know that we regularly use Natural Language Processing to get insights into topical subjects ranging from the US Presidential Election to the Super Bowl ad battle. In this post, we thought it would be interesting to collect and analyze Tweets about airlines to see how passengers use Twitter as a platform to voice their opinion. We wanted to compare how often some of the better known airlines are mentioned by travelers on Twitter, what the general sentiment of those mentions were, and and how people’s sentiment varied when they were talking about different aspects of air travel.

Collecting Tweets

We chose five airlines, gathered 25,000 of the most recent Tweets mentioning them (from Friday, June 9). We chose the most recent Tweets in order to get a snapshot of what people were talking about in Tweets at any given time.

Airlines

The airlines we chose were:

  1. American Airlines – the largest American airline
  2. Lufthansa – the largest European airline
  3. Ryanair – a low-fares giant that is always courting publicity
  4. United Airlines – an American giant that is always (inadvertently) courting publicity
  5. Aer Lingus – naturally (we’re Irish).

Analysis

We’ll cover the following analyses:

  • Volume of tweets and mentions
  • Document-Level Sentiment Analysis
  • Aspect-based Sentiment Analysis

Tools used

Sentiment Analysis

Sentiment analysis, also known as opinion mining, allows us to use computers to analyze the sentiment of a piece of text. Essentially analyzing the sentiment of text allows us to get an idea of whether a piece of text is positive, negative or neutral.

For example, below is a chart showing the sentiment of Tweets we gathered that mentioned our target airlines.

This chart shows us a very high-level summary of people’s opinions towards each airline. You can see that the sentiment is generally more negative than positive, particularly in the case of the two US-based carriers, United and American. We can also see that negative Tweets account for a larger share of Ryanair’s Tweets than any other airline. While this gives us a good understanding of the public’s opinion about these certain airlines at the time we collected the tweets, it actually doesn’t tell us much about what exactly people were speaking positively or negatively about.

Aspect-based Sentiment Analysis digs in deeper

So sentiment analysis can tell us what the sentiment of a piece of text is. But text produced by people usually talks about more than one thing and often has more than one sentiment. For example, someone might write that they didn’t like how a car looked but did like how quiet it was, and a document-level sentiment analysis model would just look at the entire document and add up whether the overall sentiment was mostly positive or negative.

This is where Aspect-based Sentiment Analysis comes in, as it goes one step further and analyzes the sentiment attached to each subject mentioned in a piece of text. This is especially valuable since it allows you to extract richer insights about text that might be a bit complicated.

Here’s an example of our Aspect-based Sentiment Analysis demo analyzing the following piece of text: “This car’s engine is as quiet as hell. But the seats are so uncomfortable!”

absa screenshot 1

It’s clear that Aspect-based Sentiment Analysis can provide more granular insight into the polarity of a piece of text but another problem you’ll come across is context. Words mean different things in different contexts – for instance quietness in a car is a good thing, but in a restaurant it usually isn’t – and computers need help understanding that. With this in mind we’ve tailored our Aspect-based Sentiment Analysis feature to recognize aspects in four industries: restaurants, cars, hotels, and airlines.

So while the example above was analyzing the car domain, below is the result of an analysis of a review of a restaurant, specifically the text “It’s as quiet as hell in this restaurant”:

absa screenshot 2

Even though the text was quite similar to the car review, the model recognized that the words expressed a different sentiment because they were mentioned in a different context.

Aspect-based Sentiment Analysis in airlines

Now let’s see what we can find in the Tweets we collected about airlines. In the airlines domain, our endpoint recognizes 10 different aspects that people are likely to mention when talking about their experience with airlines.

absa airlines domain

Before we look at how people felt about each of these aspects, let’s take a look at which aspects they were actually talking about the most.

Noise is a big problem when you’re analyzing social media content. For instance when we analyzed our 25,000 Tweets, we found that almost two thirds had no mention of the aspects we’ve listed above. These Tweets mainly focused on things like online competitions, company marketing material or even jokes about the airlines. When we filtered these noisy Tweets out, we were left with 9,957 Tweets which mentioned one or more aspects.

The chart below shows which of the 10 aspects were mentioned the most.

On one hand it might come as a surprise to see aspects like food and comfort mentioned so infrequently – when you think about people giving out about airlines you tend to think of them complaining about food or the lack of legroom. On the other hand, it’s no real surprise to see aspects like punctuality and staff mentioned so much.

You could speculate that comfort and food are pretty standard across airlines (nobody expects a Michelin-starred airline meal), but punctuality can vary, so people can be let down by this (when your flight is late it’s an unpleasant surprise, which you would be more likely to Tweet about).

What people thought about each airline on key aspects

Now that we know what people were talking about, let’s take a look at how they felt. We’re going to look at how each airline performed on four interesting aspects:

  1. Staff – the most-mentioned aspect;
  2. Punctuality – to see which airline receives the best and worst sentiment for delays;
  3. Food – infrequently mentioned but a central part of the in-flight experience;
  4. Luggage – which airline gets the most Tweets about losing people’s luggage?

Staff

We saw in the mentions graph above that people mentioned staff the most when tweeting about an airline. You can see from the graph below that people are highly negative about airline staff in general, with a fairly equal level of negativity towards each airline except Lufthansa, which actually receives more positive sentiment than negative.


Punctuality

People’s second biggest concern was punctuality, and you can see below that the two US-based airlines score particularly bad on this aspect. Also, it’s worth noting that while Ryanair receives very negative sentiment in general, people complain about Ryanair’s punctuality less than any of the other airlines. This isn’t too surprising considering their exemplary punctuality record is one of their major USPs as an airline and something they like to publicize.


Food

We all know airline food isn’t the best, but when we looked at the sentiment about food in the Tweets, we found that people generally weren’t that vocal about their opinions on plane food. Lufthansa receives the most positive sentiment about this aspect, with their pretty impressive culinary efforts paying off. However it’s an entirely different story when it comes to the customer reaction towards United’s food, none of us have ever flown United here in the AYLIEN office, so from the results we got we’re all wondering what they’re feeding their passengers now.


Luggage

The last aspect that we compared across the airlines was luggage. When you take a look at the sentiment here, you can see that again Lufthansa perform quite well, but in this one Aer Lingus fares pretty badly. Maybe leave your valuables at home next time you fly with Ireland’s national carrier.

Ryanair and Lufthansa compared

So far we’ve shown just four of the 10 aspects our Aspect-based Sentiment Analysis feature analyzes in the airlines domain. To show all of them together, we decided to take two very different airlines and put them side by side to see how people’s opinions on each of them compared.

We picked Ryanair and Lufthansa so you can compare a “no frills” budget airline that focuses on short-haul flights, with a more expensive, higher-end offering and see what people Tweet about each.

First, here’s the sentiment that people showed towards every aspect in Tweets that mention Lufthansa.

Below is the same analysis of Tweets that mention Ryanair.

You can see that people express generally more positive sentiment towards Lufthansa than Ryanair.  This is no real surprise since this is a comparison of a budget airline with a higher-end competitor, and you would expect people’s opinions to differ on things like food and flight experience.

But it’s interesting to note the sentiment was actually pretty similar towards the two core aspects of air travel – punctuality and value.

The most obvious outlier here is the overwhelmingly negative sentiment about entertainment on Ryanair flights, especially since there is no entertainment on Ryanair flights. This spike in negativity was due to an incident involving drunk passengers on a Ryanair flight that was covered by the media on the day we gathered our Tweets, skewing the sentiment in the Tweets we collected. These temporary fluctuations are a problem inherent in looking at snapshot-style data samples, but from a voice-of-the-customer point of view they are certainly something an airline needs to be aware of.

This is just one example of how you can use our Text Analysis API to extract meaning from content at a large scale. If you’d like to use AYLIEN to extract insights from any text you have in mind, click on the image at the end of the post to get free access to the API and start analyzing your data. With the extensive documentation and how-to blogs, as well as detailed tutorials and a great customer support, you’ll have all the help you’ll need to get going in no time!





Text Analysis API - Sign up




0

For the next instalment of our monthly media roundup using our News API, we thought we’d take a look at the content that was shared most on social media in the month of May. Finding out what content performs well on each social network gives us valuable insights into what media people are consuming and how this varies across different networks. To get these insights, we’re going to take a look at the most-shared content on Facebook, LinkedIn and Reddit.

Together, the stories we analyzed for this post were shared over 10 million times last month. Using the News API, we can easily extract insights about this content in a matter of minutes. With millions of new stories added every month in near real-time, News API users can analyze news content at any scale for whatever topic they want to dig into.

Most Shared Stories on Each Social Network

Before we jump into all of this content, let’s take a quick look at what the top three most-shared stories on each social network were. Take particular note of the style of articles and the subject matter of each article and how they differ across each social network.

Most shared stories on Facebook in May

  1. Drowning Doesn’t Look Like Drowning,” Slate, 1,337,890 shares.
  2. This “All About That Bass” Cover Will Make Every Mom Crack Up,” Popsugar, 913,768 shares.
  3. Why ’80s Babies Are Different Than Other Millennials,” Popsugar, 889,788 shares.

1

Most shared stories on LinkedIn in May

  1. 10 Ways Smart People Stay Calm,” Huffington Post UK, 8,398 shares.
  2. Pepsi Turns Up The Heat This Summer With Release Of Limited-Edition Pepsi Fire,” PR Newswire, 7,769 shares.
  3. In Just 3 Words, LinkedIn’s CEO Taught a Brilliant Lesson in How to Find Great People,” Inc.com, 7,389 shares.

2

Most shared stories on Reddit in May:

  1. Trump revealed highly classified information to Russian foreign minister and ambassador,” The Washington Post, 146,534 upvotes.
  2. Macron wins French presidency by decisive margin over Le Pen,” The Guardian, 115,478 upvotes.
  3. Youtube family who pulled controversial pranks on children lose custody,” The Independent, 101,153 upvotes.

3

Content Categories

Even from the article titles alone, you can already see there is a difference between the type of stories that do well on each social network. Of course it’s likely you already knew this if you’re active on any of these particular social networks. To start our analysis, we decided to try and quantify this difference by gathering the most-shared stories on each network and categorizing them automatically using our News API to look for particular insights.

From this analysis, you can see a clear difference in the type of content people are more likely to share on each network.

LinkedIn

LinkedIn users predictably share a large amount of career-focused content. However, more surprisingly stories which fall into the Society category were also very popular on LinkedIn.

Most-shared stories by category on LinkedIn in May

Reddit

Reddit is a content-sharing website that has a reputation for being a place where you can find absolutely anything, especially more random, alternative content than you would find on other social media. So it might come as a bit of a surprise to see that over half of the most-shared content on Reddit falls into just two categories, Politics and News.

Most-shared stories by category on Reddit in May


Facebook

Not surprisingly our analysis, as shown in the pie chart below, shows that almost half of the most-shared stories on Facebook are about either entertainment or food.

Most-shared stories by category on Facebook in May


Note: As a reminder we’ve only analyzed the most shared, liked and upvoted content on each platform.

Topics and Keywords

So far we’ve looked at what categories the most shared stories fall into across each social channel, but we also wanted to dig a little deeper into the topics they discussed in order to understand what content did better on each network. We can do this by extracting keywords, entities and concepts that were mentioned in each story and see which were mentioned most. When we do this, you can see a clear difference between the topics people share on each network.

LinkedIn

Below, you can see the keywords from the most shared stories on LinkedIn. These keywords are mostly business-focused, which validates what we found with the categories feature above.

Keywords extracted from the most-shared stories on LinkedIn in May

Reddit

Likewise with Reddit, you can see below that the keywords validate what the categorization feature found – that most of the content is about politics and news.

Keywords extracted from the most-shared stories on Reddit in May

Facebook

However on Facebook the most popular content tends to include mentions of family topics, like “father” and “kids,” and “baby” (with the obligatory mentions of “Donald Trump,” of course). This doesn’t correspond with what we found when we looked at what categories the stories belonged to – Arts & Entertainment and Food made up almost 50% of the most-shared content. Take a look below at what keywords appeared most frequently in the most-shared content.

Keywords extracted from the most-shared stories on Facebook in May

In order to find out why there wasn’t as clear a correlation between keywords and categories like we saw on the other platforms, we decided to dive into where this most shared content on Facebook was coming from. Using the source domain feature on the stories endpoint, we found that over 30% of the most shared content was published by one publication – Popsugar. Popsugar, for those who don’t know, is a popular lifestyle media publisher whose content is heavily weighted towards family oriented content with a strong celebrity slant. This means a lot of the content published on Popsugar could be categorized as Arts and Entertainment, while also talking about families.

Most-shared stories by source on Facebook in May

Content Length

After we categorized the stories and analyzed what topics they discuss, we also thought it might be interesting to understand what type of content, long-form or short-form, performs best across each platform. We wanted to see if the length of an article is a good indicator of how content performs on a social network. Our guess was that shorter pieces of content might perform best on Facebook while longer articles would most likely be more popular on LinkedIn. Using the word count feature on the histograms endpoint, it’s extremely easy to understand the the relationship between an article’s popularity and it’s length.

For example, below you can see that the content people shared most on Facebook was usually between 0 and 100 words in length, with people sharing longer posts on LinkedIn and Reddit.

Word count of the most-shared stories on each platform

Conclusions

So to wrap up, we can come to some conclusions about what content people shared in May:

  1. People shared shorter, family-oriented and lighthearted content on Facebook;
  2. Longer, breaking news content involving Donald Trump dominated Reddit;
  3. On LinkedIn, people shared both short and long content that mainly focused on career development and companies.

If you’d like to try the News API out for yourself, click on the image below to start your free 14-day trial, with no credit card required.




News API - Sign up




0

Artificial Intelligence and Machine Learning play a bigger part in our lives today than most people can imagine. We use intelligent services and applications every day that rely heavily on Machine Learning advances. Voice activation services like Siri or Alexa, image recognition services like Snapchat or Google Image Search, and even self driving cars all rely on the ability of machines to learn and adapt.

If you’re new to Machine Learning, it can be very easy to get bogged down in buzzwords and complex concepts of this dark art. With this in mind, we thought we’d put together a quick introduction to the basics of Machine Learning and how it works.

Note: This post is aimed at newbies – if you know a Bayesian model from a CNN, head on over to the research section of our blog, where you’ll find posts on more advanced subjects.

So what exactly is Machine Learning?

Machine Learning refers to a process that is used to train machines to imitate human intuition – to make decisions without having been told what exactly to do.

Machine Learning is a subfield of computer science, and you’ll find it defined in many ways, but the simplest is probably still Arthur Samuel’s our definition from 1959: “Machine Learning gives computers the ability to learn without being explicitly programmed”. Machine Learning explores how programs, or more specifically algorithms, learn from data and make predictions based on it. These algorithms differ from traditional programs by not relying on strict coded instruction, but by making data-driven, informed predictions or decisions based on sample training inputs. Its applications in the real world are highly varied but the one common element is that every Machine Learning program learns from past experience in order to make predictions in the future.

Machine Learning can be used to process massive amounts of data efficiently, as part of a particular task or problem. It relies on specific representations of data, or “features” in order to recognise something, similar to how when a person sees a cat, they can recognize it from visual features like its shape, its tail length, and its markings, Machine Learning algorithms learn from from patterns and features in data previously analyzed.

Different types of Machine Learning

There are many types of Machine Learning programs or algorithms. The most common ones can be split into three categories or types:

    1. Supervised Machine Learning
    2. Unsupervised Machine Learning
    3. Reinforcement Learning

1. Supervised Machine Learning

Supervised learning refers to how a Machine Learning application has been trained to recognize patterns and features in data. It is “supervised”, meaning it has been trained or taught using correctly labeled (usually by a human) training data.

The way supervised learning works isn’t too different to how we learn as humans. Think of how you teach a child: when a child sees a dog, you point at it and say “Look! A dog!”. What you’re doing here essentially is labelling that animal as a “dog”. Now, It might take a few hundred repetitions, but after a while the child will see another dog somewhere and say “dog,” of their own accord. They do this by recognising the features of a dog and the association of those features with the label “dog” and a supervised Machine Learning model works in much the same way.

It’s easily explained using an everyday example that you have certainly come across. Let’s consider how your email provider catches spam. First, the algorithm used is trained on a dataset or list of thousands of examples of emails that are labelled as “Spam” or “Not spam”. This dataset can be referred to as “training data”. The “training data” allows the algorithm to build up a detailed picture of what a Spam email looks like. After this training process, the algorithm should be able to decide what label (Spam or Not spam) should be assigned to future emails based on what it has learned from the training set. This is a common example of a Classification algorithm – a supervised algorithm trained on pre-labeled data.

Screenshot (58)
Training a spam classifier

2. Unsupervised Machine Learning

Unsupervised learning takes a different approach. As you can probably gather from the name, unsupervised learning algorithms don’t rely on pre-labeled training data to learn. Alternatively, they attempt to recognize patterns and structure in data. These patterns recognized in the data can then be used to make decisions or predictions when new data is introduced to the problem.

Think back to how supervised learning teaches a child how to recognise a dog, by showing it what a dog looks like and assigning the label “dog”. Unsupervised learning is the equivalent to leaving the child to their own devices and not telling them the correct word or label to describe the animal. After a while, they would start to recognize that a lot of animals while similar to each other, have their own characteristics and features meaning they can be grouped together, cats with cats and dogs with dogs. The child has not been told what the correct label is for a cat or dog, but based on the features identified they can make a decision to group similar animals together. An unsupervised model will work in the same way by identifying features, structure and patterns in data which it uses to group or cluster similar data together.

Amazon’s “customers also bought” feature is a good example of unsupervised learning in action. Millions of people buy different combinations of books on Amazon every day, and these transactions provide a huge amount of data on people’s tastes. An unsupervised learning algorithm analyzes this data to find patterns in these transactions, and returns relevant books as suggestions. As trends change or new books are published, people will buy different combinations of books, and the algorithm will adjust its recommendations accordingly, all without needing help from a human. This is an example of a clustering algorithm – an unsupervised algorithm that learns by identifying common groupings of data.

Screenshot (40)
Clustering visualization

Supervised Versus Unsupervised Algorithms

Each of these two methods have their own strengths and weaknesses, and where one should be used over the other is dependent on a number of different factors:
The availability of labelled data to use for training

    Whether the desired outcome is already known
    Whether we have a specific task in mind or we want to make a program for very general use
    Whether the task at hand is resource or time sensitive

Put simply, supervised learning is excellent at tasks where there is a degree of certainty about the potential outcomes, whereas unsupervised learning thrives in situations where the context is more unknown.

In the case of supervised learning algorithms, the range of problems they can solve can be constrained by their reliance on training data, which is often difficult or expensive to obtain. In addition, a supervised algorithm can usually only be used in the context you trained it for. Imagine a food classifier that has only been trained on pictures of hot dogs – sure it might do an excellent job at recognising hotdogs in images, but when it’s shown an image of a pizza all it knows is that that image doesn’t contain a hotdog.


The limits of supervised learning – HBO’s Silicon Valley


Unsupervised learning approaches also have many drawbacks: they are more complex, they need much more computational power, and theoretically they are nowhere near as understood yet as supervised learning. However, more recently they have been at the center of ML research and are often referred to as the next frontier in AI. Unsupervised learning gives machines the ability to learn by themselves, to extract information about the context you put them in, which essentially, is the core challenge of Artificial Intelligence. Compared with supervised learning, unsupervised learning offers a way to teach machines something resembling common sense.

3. Reinforcement Learning

Reinforcement learning is the third approach that you’ll most commonly come across. A reinforcement learning program tries to teach itself accuracy in a task by continually giving itself feedback based on its surroundings, and continually updating its behaviour based on this feedback. Reinforcement learning allows machines to automatically decide how to behave in a particular environment in order to maximize performance based off ‘reward‘ feedback or a reinforcement signal. This approach can only be used in an environment where the program can take signals from its surroundings as positive or negative feedback.

Reinforcement Learning in action


Imagine you’re programming a self-driving car to teach itself to become better at driving. You would program it to understand certain actions – like going off the road for example – is bad by providing negative feedback as a reinforcement signal. The car will then look at data where it went off the road before, and try to avoid similar outcomes. For instance, if the car sees a pattern like when it didn’t slow down at a corner it was more likely to end up driving off the road, but when it slowed down this outcome was less likely, it would slow down at corners more.

Conclusion

So this concludes our introduction to the basics of Machine Learning. We hope it provides you with some grounding as you try to get familiar with some of the more advanced concepts of Machine Learning. If you’re interested in Natural Language Processing and how Machine Learning is used in NLP specifically, keep an eye on our blog as we’re going cover how Machine Learning has been applied to the field. If you want to read some in-depth posts on Machine Learning, Deep Learning, and NLP, check out the research section of our blog.





Text Analysis API - Sign up




0

PREVIOUS POSTSPage 1 of 2NO NEW POSTS