Get old Tweets with Twitter4j

hellasbored

Hey guys,

so I wanna use Processing and Twitter4j to gather some serious amounts of tweets for a project. I did some research via Google and found out that Twitter now (since summer 2013) allows you to search for older tweets. Testing that on https://twitter.com/search-home everything went as expected. But when I implement the search string e.g. "digital art until:2013-06-01" I get zero results. Couldn't find any solutions yet.

The code: http://codeviewer.org/view/code:3ba1

_vk

I think in twitter4j you should not build the parameters of the search by building a string like in a regular html request. You should use Query methods instead. In this case i think since() is the one... Check the javaDoc

hellasbored

I also checked that and just tested it with query.until("2013-06-01") and query.setUntil("2013-06-01") , but same results ...

akiersky

You could try using paging:

http://twitter4j.org/en/code-examples.html (#7)

and check the last date each time until you reach your target.

This post talks about grabbing tweets from before a certain time: http://stackoverflow.com/questions/11969227/twitter4j-download-tweets-after-a-certain-tweet-point-in-time

hope this helps! ak

_vk

I think this might be an API limit, see this:

The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.

As you, I got since and until working, but only for a few days... Tested today jan 25, and got only from jan 18 to now... 8 days...

_vk

Maybe, this can help...

http://topsy.com/tweets

or...

http://topsy.com/s?q=digital art&window=a&type=tweet&sort=-date

hellasbored

Thanks for the help guys!

Topsy kinda works, but i don't know how to get the topsy results through processing to use them.

What i wanna do is search a bunch of tweets from years ago and show on a world map where users tweeted a certain hash tag over a month or so.

I thought i could get nice data from twitter, since i'll get a lot of locations too, but i don't know if my plan will work out since it seems to be very complicated.

Maybe you have some suggestions where i can find such data online? (search for certain key words + location)

_vk

I think that's only possible by buying some data... You can do this with current stream though, in present time... Those are the resellers listed in their site.

https://dev.twitter.com/programs/twitter-certified-products/products#Certified-Data-Products