Twitter Stream API Implementation?

rumlac

Hello,

I was just wondering if anyone has a working example of a processing sketch that uses the Twitter Stream API?

The few examples I can find are either out of date or not working.

Thanks!

krashe1313

I'd be curious about this too

_vk

I'll post something later. When I get sober :)

krashe1313

Hey why wait? Drunk coding could be interesting! But seriously, thanks in advance _vk.

_vk

Well... not if chars are spinning :P

Here an example. There is no rate limit check though. It will only be a problem if you filter for very popular stuff. I could not find my code to deal with it, and got no time to re do it no. But is not really a big deal. ( you see drunk code could be better than hangover code... Rate limits are for the serach API, i guess...)

import twitter4j.util.*;
import twitter4j.*;
import twitter4j.management.*;
import twitter4j.api.*;
import twitter4j.conf.*;
import twitter4j.json.*;
import twitter4j.auth.*;

TwitterStream twitterStream;


void setup() {     
  size(100, 100);    
  background(0); 
  openTwitterStream();
}  


void draw() {     
  background(0);
}  




// Stream it
void openTwitterStream() {  

  // OAuth stuff
  ConfigurationBuilder cb = new ConfigurationBuilder();  
  cb.setOAuthConsumerKey("FILL");
  cb.setOAuthConsumerSecret("FILL");
  cb.setOAuthAccessToken("FILL");
  cb.setOAuthAccessTokenSecret("FILL"); 

  //the stream object
  TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();

  // filter is used to pass querys to the stream
  // see twitter4j's java doc
  FilterQuery filtered = new FilterQuery();

  // if you enter keywords here it will filter, otherwise it will sample
  String keywords[] = {
    "hangover"
  };

  //track is like "search"... well kind of
  // for a better explanation go 
  // dev.twitter.com/streaming/overview/request-parameters"
  filtered.track(keywords);

  // the StatusListener interface is where the magic happens
  // code there will be executed upon tweets arriving
  // so we want to attach one to our stream
  twitterStream.addListener(listener);

  if (keywords.length==0) {
    // sample() method internally creates a thread which manipulates TwitterStream 
    // and calls these adequate listener methods continuously.
    // ref //dev.twitter.com/streaming/reference/get/statuses/sample
    // "Returns a small random sample of all public statuses"
    twitterStream.sample();
  } else { 
    twitterStream.filter(filtered);
  }
  println("connected");
} 


// Implementing StatusListener interface
// the methods are pretty much self explantory
// they will be called according to the messages arrived
// onStatus is probabbly what you are lookikng for...
StatusListener listener = new StatusListener() {

  //@Override
  public void onStatus(Status status) {
    System.out.println("@" + status.getUser().getScreenName() + " - " + status.getText());
  }

  //@Override
  public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
    System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
  }

  //@Override
  public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
    System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
  }

  //@Override
  public void onScrubGeo(long userId, long upToStatusId) {
    System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
  }

  //@Override
  public void onStallWarning(StallWarning warning) {
    System.out.println("Got stall warning:" + warning);
  }

  //@Override
  public void onException(Exception ex) {
    ex.printStackTrace();
  }
};

rumlac

Hi, so I have tried running your code but I get the message: "Cannot find a class or type named "TwitterStream".

Thanks

_vk

Hello. the code is working here with processing 2.2.1 and tw4j 4.0.3.

Probably some change in tw4j API... We gota check...

_vk

Ok, now tested with latest stabel version tw4j 4.0.4 (still processing 2.2.1). And its working fine. Also the java doc makes no mention to changes in TwitterStream interface. So maybe you have a problem in your import. Perhaps you have not imported twitter4j-stream-x.x.x?

Or ir can be a processing 3 issue, but i think it's not likely.

krashe1313

I'm going to have to study this and try it tomorrow. Big thanks!

rumlac

_vk, it is probably an issue with my import. I have started again by placing the folder "twitter4j-4.0.4" into my processing libraries folder, I'm not exactly sure which files from the folder to add to the sketch.

_vk

Installing a java library is weird. there is this article:

https://github.com/processing/processing/wiki/How-to-Install-a-Contributed-Library

Last stuff is about non processing libraries.

The way I did, and is working, is weird and involves changing the names of the jars it selfs. As I get a complaint about - and . in the names. The final result is like this (those are inside mysketchbook/libraries):

note the name of the folder is changed, all jars got rid os dashes and point and 'lib' folder is renamed to library.

Alternatively you can just drop the folder in your sketch in PDE and it will work ONLY for that sketch, which is not optimal.

rumlac

Thanks _vk, That has solved the previous issue. However, I am now getting the message:

"The type new StatusListener(){} must implement the inherited abstract method StatusListener.onStatus(Status)"

I am unsure what this is referring to.

Thanks again.

GoToLoop

"The type new StatusListener() {} must implement the inherited abstract method StatusListener.onStatus(Status)"

abstract methods are merely some description about what are their parameters' types and their return type. In other words, they are empty methods! :|

In order for a non-abstract class to inherit from a class or interface containing abstract methods,
the former must implement each of them.

It's also highly advisable to prefix those method implementations w/ @ Override.
So the Java compiler can check whether the implementation matches the abstract method.

Take a look at @_vk's example for a StatusListener's anonymous instantiation. *-:)

_vk

@GoToLoop do you understand this importing non processing libraries? Could you help understanding this weird need to change the jar files names?

GoToLoop

My only experience was once importing a library w/ 1 ".jar" file only. :|

_vk

Ok thanks anyway.

REAS

Temboo makes it easier to interface with Twitter. You loose some control for ease of use. Depending on how much data you need to request, it's either free or it requires a fee: https://www.temboo.com/processing

_vk

Also if using p5.js there is a amazing tutorial by Daniel Shiffman using a js library, I think is tweet. Worth watching.

Actually he shows a way you could even use a java code (twitter4j) and deploy it via node and command line.

_vk

@REAS you probably could say if the method of importing twitter4j described above is correct, couldn't you. :)

rumlac

Looking at Temboo, would I be right in thinking that it uses the search API rathe than the streaming API and can therefore not be left running for a long period of time due to the limits? I am simply trying to find a way to receive tweets with a certain (unpopular) hashtag, even if it is just 1 tweet per minute rather than all tweets with in real time. I'm still not having any luck. Thanks again

_vk

I'm still not having any luck.

Why? the code I posted above does exactly that. What is going wrong? Still the same problem? @GoToLoop have told you how to go about the

"The type new StatusListener(){} must implement the inherited abstract method StatusListener.onStatus(Status)"

Just implement all methods, even if you just use some of them...

rumlac

Yes, i'm still having trouble with that part. I'm not exactly sure what is meant by 'implement all methods' or how to go about doing so.

Thanks again

_vk

In this case you should have asked here ;)

just use this (copied from the post above)

StatusListener listener = new StatusListener() {

  //@Override
  public void onStatus(Status status) {
    System.out.println("@" + status.getUser().getScreenName() + " - " + status.getText());
  }

  //@Override
  public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
    System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
  }

  //@Override
  public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
    System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
  }

  //@Override
  public void onScrubGeo(long userId, long upToStatusId) {
    System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
  }

  //@Override
  public void onStallWarning(StallWarning warning) {
    System.out.println("Got stall warning:" + warning);
  }

  //@Override
  public void onException(Exception ex) {
    ex.printStackTrace();
  }
};

Whatever you want to do with the status arrived you do inside the onStatus(Status status) method, referring to the arrived Status (a tweet) as status. For instance you want to display the text from that status you would say

StatusListener listener = new StatusListener() {

 //do your stuff here..
//@Override
      public void onStatus(Status status) {
        String tweet = status.getText();
        text(tweet, 10, 100);
      }

  // Keep those methods here... this is it...
  //@Override
  public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
    System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
  }

  //@Override
  public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
    System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
  }

  //@Override
  public void onScrubGeo(long userId, long upToStatusId) {
    System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
  }

  //@Override
  public void onStallWarning(StallWarning warning) {
    System.out.println("Got stall warning:" + warning);
  }

  //@Override
  public void onException(Exception ex) {
    ex.printStackTrace();
  }
};

But keep all other methods. even if they are empty..

Each one is called on each event, e.g. if someone delete an status (tweet) you should delete it from your database (if you have stored it). The code to do this would go inside onDeletionNotice(StatusDeletionNotice statusDeletionNotice) method. So it would be called whenever a deletion notice arrives. is that clear?

Post your code if you need further assistance.

rumlac

Hi _vk,

I haven't changed anything in the code so here is what I am trying to run (minus OAuth bits) :

import twitter4j.util.*;
import twitter4j.*;
import twitter4j.management.*;
import twitter4j.api.*;
import twitter4j.conf.*;
import twitter4j.json.*;
import twitter4j.auth.*;

TwitterStream twitterStream;


void setup() {     
  size(100, 100);    
  background(0); 
  openTwitterStream();
}  


void draw() {     
  background(0);
}  


// Stream it
void openTwitterStream() {  

  // OAuth stuff
  ConfigurationBuilder cb = new ConfigurationBuilder();  
  cb.setOAuthConsumerKey("");
  cb.setOAuthConsumerSecret("");
  cb.setOAuthAccessToken("");
  cb.setOAuthAccessTokenSecret(""); 

  //the stream object
  TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();

  // filter is used to pass querys to the stream
  // see twitter4j's java doc
  FilterQuery filtered = new FilterQuery();

  // if you enter keywords here it will filter, otherwise it will sample
  String keywords[] = {
    "hangover"
  };

  //track is like "search"... well kind of
  // for a better explanation go 
  // dev.twitter.com/streaming/overview/request-parameters"
  filtered.track(keywords);

  // the StatusListener interface is where the magic happens
  // code there will be executed upon tweets arriving
  // so we want to attach one to our stream
  twitterStream.addListener(listener);

  if (keywords.length==0) {
    // sample() method internally creates a thread which manipulates TwitterStream 
    // and calls these adequate listener methods continuously.
    // ref //dev.twitter.com/streaming/reference/get/statuses/sample
    // "Returns a small random sample of all public statuses"
    twitterStream.sample();
  } else { 
    twitterStream.filter(filtered);
  }
  println("connected");
} 


// Implementing StatusListener interface
// the methods are pretty much self explantory
// they will be called according to the messages arrived
// onStatus is probably what you are lookikng for...
StatusListener listener = new StatusListener() {

 //do your stuff here..
//@Override
      public void onStatus(Status status) {
        String tweet = status.getText();
        text(tweet, 10, 100);
      }

  // Keep those methods here... this is it...
  //@Override
  public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
    System.out.println("Got a status deletion notice id:" + statusDeletionNotice.getStatusId());
  }

  //@Override
  public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
    System.out.println("Got track limitation notice:" + numberOfLimitedStatuses);
  }

  //@Override
  public void onScrubGeo(long userId, long upToStatusId) {
    System.out.println("Got scrub_geo event userId:" + userId + " upToStatusId:" + upToStatusId);
  }

  //@Override
  public void onStallWarning(StallWarning warning) {
    System.out.println("Got stall warning:" + warning);
  }

  //@Override
  public void onException(Exception ex) {
    ex.printStackTrace();
  }
};

Thanks

_vk

I can't run it now. I can check it later. Is this working? Do you get any messages? If so, which? In which line?

rumlac

No, it still returns the message "The type new StatusListener(){} must implement the inherited abstract method StatusListener.onStatus(Status)" highlighting the line: StatusListener listener = new StatusListener() {

Thanks

_vk

Well that's weird. I copy the code, insert oAuth data, hit run, and... Everything goes fine, getting tweets about hangover

A note. the way I did in line 79 in your code, will fail to print the tweet to the canvas, as the background call in draw() will erase it immediately and the text is write outside the canvas. You may change size in setup() and get rid of background() or, better, just keep the original println(tweet) while debugging.

I really don't see where the problem is, as the code runs perfect here...

rumlac

Could it be an issue with the files in the sketch folder?

_vk

As I said, I don't know. The error does not look like is an import error, I guess, as it is complaining about the implementation of the interface.

But you could try to just drop in a NEW sketch the extracted twitter4j jars (the files inside lib folder, but the readme) - without changing anything - then copy/paste the code an run.

The above about importing, with all the changes I made, is NOT to be made when you use the sketch's code folder. It's meant to have the library in processing's libraries folder and just use import in the code.

The reason I'm suggesting the new sketch is to reassure there is no other silly errors, like missing brackets or whatever. As I see in the same sketch you got a lot more libraries.

Try this virgin sketch with droped code

GoToLoop

Placing ".jar" files inside "code/" subfolder should be a last ditch resort!
Strive to get your libraries from the PDE's "Contribution Manager"!

rumlac

I would do that but when I try to add the library from within processing, it doesn't appear on the list of available libraries

rumlac

Success! Creating a new sketch as you advised did the job. Thanks for you help!

_vk

Good to hear. #:-S

Placing ".jar" files inside "code/" subfolder should be a last ditch resort!

Well that was a last ditch resort... Just for debugging

Strive to get your libraries from the PDE's "Contribution Manager"!

This is a non Processing library. A java library.

GoToLoop

This is a non Processing library. A java library.

Oh, I didn't know. Even so, it's much preferable to force it to install along Processing's libraries so it's available for all sketches. :ar!