querying github API for users with README that matches text -


i retrieve users repositories contain readme file contains text matched string passed in query. possible using github api?

in addition, include location , language in query.

thanks.

this not straightforward using available api now. however, can use api want.

be warned there on 10 million repositories on github - take long time. can retrieve list of 100 repositories per query, need use pagination -> more 100000 requests repositories. user limited 5000 requests per hour, "banned" hour. take more 40 hours, if you're using 1 user credentials.

steps:

  1. get json repositories (https://developer.github.com/v3/repos/#list-all-public-repositories)

  2. use pagination fetch 100 objects per query (https://developer.github.com/v3/#link-header)

  3. decode json , retrieve list of repositories

  4. for each repository need repository url object json, gives link repository.

  5. now need readme contents. there 2 ways : a) use github api, using repo url , sending request : https://api.github.com/repos/:owner/:repo/readme( https://developer.github.com/v3/repos/contents/#get-the-readme) , either decode file (it encoded using base64) or follow html property of json e.g "html": "https://github.com/pengwynn/octokit/blob/master/readme.md". if there no readme, 404 not found code, can proceed next repository.

    b) make url readme using step 4 gives e.g. https://api.github.com/repos/octocat/hello-world ; , parse , transform https://github.com/octocat/hello-world/readme.md ; more complicated, in case there no readme.

  6. search through file specific text, , record or not if have found text.

  7. iterate until went through repositories.

advanced things - if plan on running more often, can recommend use caching https://developer.github.com/v3/#conditional-requests ; store date + time when have done query, , use later see if has changed in repository. eliminate many of subsequent queries if need have up-to-date information. still have retrieve whole list of repositories though. (but search updated repositories)

of course make faster, can improve algorithm make parallel - retrieve 100 repositories, proceed retrieve next 100, , in meanwhile search if first 100 repositories contain readme file , if readme has searching for, , on. make things faster, certainly. need use sort of buffer, not know terminates faster (getting repositories list, or searching through them)

hope helps.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque...
Read more

jsf - How to ajax update an item in the footer of a PrimeFaces dataTable? -

this visual of table have: +---------+------------+-------------------+ | header1 | header2 | header3 | +---------+------------+-------------------+ | row | input | calc'ed output | | row b | input b | calc'ed output b | | etc.. | etc.. | etc.. | +---------+------------+-------------------+ | total: total calc'ed output | +------------------------------------------+ and stripped-down code: <p:datatable id="mytable" value="#{mybean.somelist}" var="currentitem"> <p:column headertext="header1"> <h:outputtext value="#{currentitem.name}" /> </p:column> <p:column headertext="header2"> <pe:inputnumber value="#{currentitem.inputval}"> <p:ajax event="change" listener="#{mybean.changelistener}" update="outputval outputtotal...
Read more

django - CSRF verification failed. Request aborted. CSRF cookie not set -

error: reason given failure: csrf cookie not set. in general, can occur when there genuine cross site request forgery, or when django's csrf mechanism has not been used correctly. post forms, need ensure: browser accepting cookies. view function uses requestcontext template, instead of context. in template, there {% csrf_token %} template tag inside each post form targets internal url. if not using csrfviewmiddleware, must use csrf_protect on views use csrf_token template tag, accept post data. you're seeing section of page because have debug = true in django settings file. change false, , initial error message displayed. can customize page using csrf_failure_view setting. did find, nothing helped, must foolish(( my template <form method="post">{% csrf_token %} {% field in form %} <div class="control-group"> {{ field }} </div> {% endfor %} <input type=...
Read more