java - Algorithm of crawling Top10 PR/Alexa sites -


i'm trying write script crawl current top 10 pr/alexa sites. since pr/alexa changes. script should take care of mean if today there not site in top 10 tomorrow.

i dont know how start with. know crawling concepts here i'm stuck. there top50 sites or top500 sites. can configure of course.

i read google spider complicated simple task. how google,yahoo,bing crawl billions of sites around web. i'm curious. cursor point, mean how google can identify newly launch site.

ok these deep details, read these later. right i'm more concern problem. how crawl top10 pr sites.

can provide sample program can understand better?

it's rather simple fetch top25sites (if understood correctly wanted do)

code:

from bs4 import beautifulsoup urllib.request import urlopen b = beautifulsoup(urlopen("http://www.alexa.com/topsites").read()) paragraphs = b.find_all('p', {'class':'desc-paragraph'}) p in paragraphs:    print(p.a.text)

output:

google.com facebook.com youtube.com yahoo.com baidu.com wikipedia.org (...)

but have in mind law in countries more strict. on own risk.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more