Throttle scraping?

Subscribe to Throttle scraping? 1 post, 1 voice

 
Avatar Paul 1 post

I’m using scRUBYt to scrape forums for a research-related search application I am creating. Many of the sites are large and I don’t want to annoy anybody by bombarding them with requests over the several hours it would take to scrape at full speed – is there any way to throttle the rate at which scRUBYt scrapes?