Fetching Tweet Metadata

Mention allows to find relevant posts on the web and social media, using complex filters including keyword-based boolean queries, keyword proximity constraints, influence scores, author country, language, etc.

However, the Twitter policies do not allow us to return the full Tweet metadata in our API (e.g. the text of tweets), like we do in our applications.

Because of this, the API will return no other metadata than the ID and URL of the Tweets.

This guide explains how to fetch the metadata of relevant Tweets found by the Mention API.

Overview

Fetching the metadata of relevant Tweets found by the Mention API can be done by using the official Twitter API. In particular, the Twitter API has a endpoint called statuses/lookup that returns the metadata of the Tweet whose IDs that are passed to it.

Register a Twitter API application

Accessing the Twitter API requires to register a Twitter API application at https://apps.twitter.com/.

On this page, click "Create New App", and follow the creation steps.

Once the app has been created, go to the "Keys and Access Tokens" section, and copy the following information:

  • Consumer Key (API Key)
  • Consumer Secret (API Secret)

Then, create an access token by clicking on the "Create my access token" button at the bottom of the page. Copy the following information:

  • Access Token
  • Access Token Secret

Accessing the Twitter API

Before this step, make sure that you retrieved the following information from the previous step:

  • Consumer Key (API Key)
  • Consumer Secret (API Secret)
  • Access Token
  • Access Token Secret

The Twitter API is a simple REST API, however it requires users to authenticate themselves by using the OAuth 1.0A protocol. Unlike OAuth 2, this version of the protocol requires a library to be used easily and safely.

Most languages have a library with good OAuth 1.0A support. Here are a few:

All of them have approximately the same workflow:

  • Create a new instance, by providing the Consumer Key (API Key), Consumer Secret (API Secret), Access Token, and Access Token Secret.
  • The instance acts like an HTTP client, which transparently authenticates the requests.

Fetching Tweet metadata

The Mention API returns Tweet IDs in the metadata.twitter.id_str attribute, in each Twitter mention.

The statuses/lookup API endpoint allows to retrieve the metadata of Tweets, by ID.

The endpoint returns Tweet metadata in the format that is documented here: https://dev.twitter.com/overview/api/tweets

Note: The endpoint is limited to 180 calls per 15 minute windows, and can fetch up to 100 tweets per call. We recommend to batch calls to the endpoint as much as possible in order to avoid calling it with less than 100 tweets. Caching tweets locally can also reduce the number of calls to the endpoint.

Examples

NodeJS

Note: This code requires the oauth npm package, which can be installed with the following command:

npm install --save oauth

var OAuth = require('oauth');

// Change these
var CONSUMER_KEY = '...';
var CONSUMER_SECRET = '...';
var ACCESS_TOKEN = '...';
var ACCESS_TOKEN_SECRET = '...';

// Instantiate the client
var twitterClient = new OAuth.OAuth(
    'https://api.twitter.com/oauth/request_token',
    'https://api.twitter.com/oauth/access_token',
    CONSUMER_KEY,
    CONSUMER_SECRET,
    '1.0A',
    null,
    'HMAC-SHA1'
);

// Request the metadata of two tweets
var IDs = ['800826466359541760', '800747234652340224'];
var URL = 'https://api.twitter.com/1.1/statuses/lookup.json?id=' + IDs.join(',');
twitterClient.get(
    URL,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET,
    function (err, data, res) {
        if (err) {
            console.log(err);
            return;
        }
        if (res.statusCode != 200) {
            console.log('Got non 200 status: ', res.statusCode);
            return;
        }
        data = JSON.parse(data);
        console.log(JSON.stringify(data, "", 2));
    }
);

Python

Note: This code requires the rauth package, which can be installed with the following command:

sudo pip install rauth

from rauth import OAuth1Service
import json

# Change these
CONSUMER_KEY = '...'
CONSUMER_SECRET = '...'
ACCESS_TOKEN = '...'
ACCESS_TOKEN_SECRET = '...'

# Instantiate a client
twitter_client = OAuth1Service(name='twitter',
                              consumer_key=CONSUMER_KEY,
                              consumer_secret=CONSUMER_SECRET,
                              base_url='https://api.twitter.com/1.1/')

twitter_session = twitter_client.get_session((ACCESS_TOKEN, ACCESS_TOKEN_SECRET))

# Request the metadata of two tweets
IDs = ['800826466359541760', '800747234652340224']
URL = 'https://api.twitter.com/1.1/statuses/lookup.json'

r = twitter_session.get(URL,
                        params={'id': ','.join(IDs)})

print(json.dumps(r.json(), indent=2))

PHP

Note: This code requires the twitteroauth package, which can be installed with the following command:

php composer require abraham/twitteroauth

<?php

require "vendor/autoload.php";

use Abraham\TwitterOAuth\TwitterOAuth;

const CONSUMER_KEY = '...';
const CONSUMER_SECRET = '...';
const ACCESS_TOKEN = '...';
const ACCESS_TOKEN_SECRET = '...';

// Instantiate the client
$twitterClient = new TwitterOAuth(
    CONSUMER_KEY,
    CONSUMER_SECRET,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET
);

// Request the metadata of two tweets
$IDs = ['800826466359541760', '800747234652340224'];
$r = $twitterClient->get('statuses/lookup', [
    'id' => implode(',', $IDs),
]);

echo json_encode($r, JSON_PRETTY_PRINT), "\n";