Jos works as Architect for JPoint. In the last couple of years Jos has worked on large projects in the public and private sector. Ranging from very technology focusses integration projects to SOA/BPM projects using WS-* and REST based architectures. Jos has given many presentations on conferences such as Javaone, NL-JUG, Devoxx etc., and has written two books for Manning: Open Source ESBs in Action and (published in the next couple of months) SOA Governance in Action. In this last book Jos shows how, with some good practical governance approaches, you can create great WS-* and REST based services and APIs. Besides this he has his own blog where he writes about interesting technologies and shares his ideas about REST, API Design, Scala, Play and more. Jos is a DZone MVB and is not an employee of DZone and has posted 51 posts at DZone. You can read more from them at their website. View Full User Profile

Access the Twitter REST API (v1.1) from Scala and Java using signpost

09.22.2012
| 5376 views |
  • submit to reddit

If you've read some other articles on this blog you might know that I like creating visualizations of various datasets. I've just started a small project where I want to visualize some data from Twitter. For this I want to retrieve information about followers and profile information directly from twitter. I actually started looking for a set of all twitter accounts, but could only find one that was two years old. So, only option left, directly access the twitter API and get the data myself.

There are a couple of open source libraries out that we can use directly from Scala (or java) but as far as I could see they use the old v1 API and not the v1.1 API. The old API has a very strict data rate limit, which is a bit lighter in the new API. And besides that I'm more interested in the raw data and parsing the returning JSON isn't that hard with Scala (or Java for that matter).

Register an application at twitter

First thing to do, and easiest way to get started is registing a new application for your twitter account. Go to https://dev.twitter.com/apps/new and create a new application. Don't worry about the urls, since we won't be using the OAuth callback mechanism:

Create 1 | Twitter Developers.png

Depending on what you want to do with the API you need to give additional permissions to this app. Default is "read-only", if you want to allow your new application to post or access your direct messages you need to update the permissions. This is done from the settings page of your application:

SmartjavaTest | Twitter Developers.png

Once you've created the application and setup the correct permissions you can generate an access token. Doing this will avoid having to go through the complete OAuth dance. You do this by going to your new app details and at the bottom select the "create my access token" option.

Create 2 | Twitter Developers.png

Now you'll have a set of tokens (see the details part of your applications):

SmartjavaTest | Twitter Developers-1.png

We'll use these tokens to authenticate the requests we'll make to twitter.

Using a OAuth library

The OAuth protocol is a pretty good documented protocol, but implementing it yourself is a lot of work and very error-prone. Luckily there are many OAuth libraries that can help you. I've tried a couple and the one that's most easy to use (at least for me) was signpost. The examples below show how to do this from Scala, but you can follow the same approach for Java.

First the dependencies. I've used sbt and from signpost I use the client with support for HTTP commons. In sbt add the following:

..
libraryDependencies ++= Seq(
	"oauth.signpost" % "signpost-core" % "1.2",
	"oauth.signpost" % "signpost-commonshttp4" % "1.2", 
	"org.apache.httpcomponents" % "httpclient" % "4.2",
         ...
)

For maven you can use the same libraries. Next we can write a simple test to see if everything is working. In java it looks like this:

import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
 
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
 
 
public class Tw {
 
	  static String AccessToken = "access token for your app";
	  static String AccessSecret = "access secret for your app";
	  static String ConsumerKey = "consumer key for your app";
	  static String ConsumerSecret = "consumer secret for your app";
 
	/**
	 * @param args
	 */
	public static void main(String[] args) throws Exception {
		OAuthConsumer consumer = new CommonsHttpOAuthConsumer(
                ConsumerKey,
                ConsumerSecret);
 
        consumer.setTokenWithSecret(AccessToken, AccessSecret);
        HttpGet request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
        consumer.sign(request);
 
        HttpClient client = new DefaultHttpClient();
        HttpResponse response = client.execute(request);
 
        int statusCode = response.getStatusLine().getStatusCode();
        System.out.println(statusCode + ":" + response.getStatusLine().getReasonPhrase());
        System.out.println(IOUtils.toString(response.getEntity().getContent()));
	}
}

And in Scala it looks pretty much the same:

import org.apache.http.client.HttpClient
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpGet
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer
import org.apache.commons.io.IOUtils
 
 
object TwitterPull {
 
	  val AccessToken = "access token for your app";
	  val AccessSecret = "access secret for your app";
	  val ConsumerKey = "consumer key for your app";
	  val ConsumerSecret = "consumer secret for your app";
 
 
  def main(args: Array[String]) {
 
	 val consumer = new CommonsHttpOAuthConsumer(ConsumerKey,ConsumerSecret);
	 consumer.setTokenWithSecret(AccessToken, AccessSecret);
 
     val request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
     consumer.sign(request);
     val client = new DefaultHttpClient();
     val response = client.execute(request);
 
     println(response.getStatusLine().getStatusCode());
     println(IOUtils.toString(response.getEntity().getContent()));
  }
}

When you run this the output will look something like this:

200
{"previous_cursor_str":"0","next_cursor":0,"ids":[48342167,21011010,824959303,97242821,16953163,218083367,20869799,5234221,13604142,804783128,271050984,405121284,26470609,50201837,1723451,374494377,120867838,14311946,253114713,39554511,7375412,42507395,112806109,92787154,218238023,110443797,76922155,198798790,294104985,305625416,217698029,21803482,14927822,15453445,15715866,15657036,186956616,36028164,70380613,326158542,573546312,14401332,521488579,9108612,576970378,293236313,16398366,16220300,15234937,32000283,439444353,14300622,67204409,155850135,14198255,32264673,15852981,313248158,20123099,608942046,234930032,36896958,18466675,45496942,330899833,18980755,88253383,461023805,31175627,11044952,142780445,63175189,107991607,94830953,600993241,6195002,115391430,550080945,381418927,168603682,142388604,8258462,218411138,30450578,77728346,2521381,182867524,494119147,29426983,572417260,94344849,325413275,389354525,501438275,164346498,22730282,8293302,21085554,341645357,56978853,180507788,10074002,22536424,14247654,581293627,15259428,483317230,462826270,4774641,15366832,96850673,278486993,22273826,17716679,14566626,158473088,20461042,161242434,43756629,40163100,141165981,5325152,7620782,266749648,524476136,557713614,39602637,18843154,1623,565954426,39639621,166672305,18683074,233118689,44876099,235258223,219310062,10699922,12660502,218030046,91552210,19361980,206645598,35346200,58440021,470388557,26495649,59066453,40292255,543375441,33242290,6015852,317150447,22935775,232300346,476045917,90913482,249088920,67658976,614873,522722520,186766721,285517705,71683175,131444964,166501605,477920664,38154550,18738205,8861832,15594932,18536741,7595202,465378842,11838952,14848133,431696576,14358671,414520167,222578501,67058139,28976735,95601387,426582611,24874129,418762594,128157235,106030956,31352215,18733178,260196778,153179029,91842580,229494512,83414433,285579699,19957600,54295155,14929418,51516573,200076011,18758733,17776895,59397841,216802709,149834999,327507356,8200322,174345369,108636400,27504001,326877592,139919716,49949338,215035403,118421144,49410665,149550914,18446431,25662335,261725134,267634174,57737391,146506056,126964949,71055234,20870640,210196418,222806923,13290742,72247756,180410163,14784480,36684216,25611502,95614691,54629161,112967594,181656257,17994312,72918901,140082918,149087212,137272324,99534020,121755576,93964779,35848342,43059008,34704029,87672717,113137792,17863333,90407665,90591814,54297023,57924897,87551006,28300354,48990752,26188013],"previous_cursor":0,"next_cursor_str":"0"}

If you get a 403 check whether the tokens match.
Published at DZone with permission of Jos Dirksen, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)