Make Sure you are logged out before you spider any site...
This tool will click on all links, including delete links... so don't be logged into the sites you're spidering.
Also - this can bring a server to it's knees (depending on what's being spidered) so here are some suggested settings:
[Settings] –
Put in a project name and a URL –
Uncheck: Respect 'robots.txt' –
Check: Exclude Images –
Add to Exclude filters: /files/* (and anything else you want to exclude) –
(others are fine as default) [Crawler] –
Request Delay: 1.0+ –
Thread Count: 3 –
(others are fine as default)
Updated version and application homepage available www.auditmypc.com