dCrawl

dCrawl is a web crawler (also known as a web spider): the purpose is to scan a website pages' links in order to get a list of files and directories hosted on the target server.

The crawler only retrieves files that are explicitly linked on the website, it will not discover new files magically. This app is only useful if you want to approximately know the kind of files hosted on a certain website or, if you are the website's owner, to know the amount of files easily reachable from the visitors.

Since dCrawl is still being developed, weird behaviors (bugs) may happen from time to time: you might find "wrong" files (e.g. files that do not exists, javascript snippets of code, and so on) among the results: you will most likely realize that some files are not actual files, but still don't get too angry when you realize that "document.location.href;" or something like that does not exist.

Other things that could happen on some website (and this is mostly the website's problem) is that, for every page that is visited, other pages are "created" (e.g. visiting /goto/fail might create a page containing a /goto/fail/goto/fail link, which then would contain a /goto/fail/goto/fail/goto/fail link and so on). When you realize that this kind of loop is going on, you can stop the execution using the "Stop" button.

One last thing: avoid scanning a huge number of files since a) the website's owner might not be happy about it and b) you might end up using all the memory available.

This being said, please keep in mind that dCrawl is still under development, hence new updates will be available every now and then.

TODO:

- Ignore javascript code when parsing a page

- Read robots.txt file (if available) to get more files and directories to work on (not really fair, but I am evil like that)

- Distinguish (maybe using a different color) non-empty directories from files and empty directories

- Export the results of a scan

- Fix the bugs you awesome people will tell me in a detailed and kind way :-)

If you enjoyed this app, please consider tipping using BTC, I would really appreciate it

18XJhDrTRt1xDHvGEEzghRQXmKR4z3qSPT