Recent Posts by Paul

Subscribe to Recent Posts by Paul 1 post found

Jul 24, 2008
Avatar Paul 1 post

Topic: Usage HOWTOs / Throttle scraping?

I’m using scRUBYt to scrape forums for a research-related search application I am creating. Many of the sites are large and I don’t want to annoy anybody by bombarding them with requests over the several hours it would take to scrape at full speed – is there any way to throttle the rate at which scRUBYt scrapes?