User lookup on Twitter API from R results in error (403) -
using twitter api , twitter
-package, trying retrieve user objects long list of names (between 50.000 , 100.000).
i keep getting following error:
error in twinterfaceobj$doapicall(paste("users", "lookup", sep = "/"), : client error: (403) forbidden
the error code supposedly hints @ "update limits". rate limit on user lookups 180 , lookups performed in batches of 100 user names. therefore 18.000 users shouldn't problem. reducing number 6000 (to respect limit on requests via application-only auth) per 15 minute time window results in same error.
here mwe (for do, however, need your own api-keys):
library(plyr) # install latest versions github: # devtools::install_github("twitter", username="geoffjentry") # devtools::install_github("hadley/httr") library(twitter) library(httr) source("twitterkeys.r") # own api-keys setup_twitter_oauth(consumerkey, consumersecret, accesstoken, accesssecret) # following generate large enough list of user names: searchterms <- c("worldcup", "economy", "climate", "wimbledon", "apple", "android", "news", "politics") # might take while sample <- llply(searchterms, function(term) { tweets <- twlisttodf(searchtwitter(term, n=3200)) users <- unique(tweets$screenname) return(users) }) usernames <- unique(unlist(sample)) # function supposed perform lookups in batches # , mind rate limit: getuserobjects <- function(users) { groups <- split(users, ceiling(seq_along(users)/6000)) userobjects <- ldply(groups, function(group) { objects <- lookupusers(group) out <- twlisttodf(objects) print("waiting 15 minutes...") sys.sleep(900) return(out) }) return(userobjects) } # putting action: userobjects <- getuserobjects(usernames)
sometimes looking smaller subsets manually e.g. via lookupusers(usernames[1:3000])
works; when try automate process, however, error gets thrown.
does have idea reason might be?
according answer i hit rate limit twitter first request , there not limits on total number of users, on number of calls per 15 minute interval. if each call has 100 users, , trying 6000 users, should making 60 calls, more 15 allowed. try putting program sleep , having send out call again after 15 minutes up.
Comments
Post a Comment