Recent Posts by Paul
|
Jul 24, 2008
|
Topic: Usage HOWTOs / Throttle scraping? I’m using scRUBYt to scrape forums for a research-related search application I am creating. Many of the sites are large and I don’t want to annoy anybody by bombarding them with requests over the several hours it would take to scrape at full speed – is there any way to throttle the rate at which scRUBYt scrapes? |