CrawlNScrape

WHAT IS CRAWLNSCRAPE?

CrawlNScrape facilitates crawling through the internet, following links from website to website, peering in here and there, getting an introduction to ethical internet crawling and HTML scraping. This is a true crawl through unfamiliar, and perhaps unknown, facets of the internet.

CrawlNScrape permits you to visit arbitrary websites to extract whatever data may be found there - technical bits such as details of the HTML code, images, icon, author, description, keywords, Meta Data, Forms Data, Media, and especially IP addresses, geographic Locations and links - and yet more especially - links to other websites!

With CrawlNScrape the web crawling is under your control. A typical web crawler such as a Google bot is given a set of “seed sites” and turned loose to crawl and scrape. With CrawlNScrape, you are the bot and CrawlNScrape is your tool for crawling and scraping. You control the choice of seed site, which sites you will visit and what data you will scrape.

If you are interested in internet crawling and website scraping you should enjoy working with this app. It can be tedious until you become familiar with how to Select | Copy | Paste on your device, how to use The Stack, until you accommodate yourself to the pace of crawling! and until you discover which websites are “good seeds” for your particular interests - preferably those with many offsite links.

ETHICAL HTML SCRAPING...

The web crawler should respect the rules set by robots.txt. CrawlNScrape gives you the tools to work this way. HTML scraping is just like any other tool - you can use it for good stuff and you can use it for bad stuff. That HTML scraping itself is not illegal doesn’t mean you can scrape any site you want. Some sites explicitly prohibit data extraction either via the robots.txt file or their Terms of Service page. CrawlNScrape gives you the tools to download and study the robots.txt file, so you can choose to visit or not visit individual sites, and to scrape or not scrape various folders and files, as appropriate.

THE DEEP WEB!

With CrawlNScrape you can collect URLs of pages where you may want to extract the HTML code and data. With Deep Crawling the idea is to search any web page for links, especially for links to other websites. Then explore those sites for further links, to other countries, to wherever. Then continue, deeper and deeper, into the World Wide Web.

GETTING STARTED...

From the opening view CrawlNScrape has practical, introductory lessons to get you started. Plus you will find that you can exit to any other app such as Google Maps, Google Search, a text editor and to your favorite browser, then return to CrawlNScrape while keeping your “breadcrumbs” intact in The Stack, so you can go wherever there is a place to go and explore whatever is to be found there, with confidence that you can get back there again.

A PREVIEW IS AVAILABLE!

This introductory Crawl begins with an overview of the CrawlNScrape menu options so you gain an understanding of the app structure and flow. It then starts a crawl at https://www.example.com in Phoenix, Arizona, United States and tours throughout the internet to Stockholm, Sweden. Afterwards, you could perhaps download this app and continue this tour through Stockholm, Sweden; London, England; Dublin, Ireland; and, well, to wherever…

… to see what you can see

FOLLOW THIS LINK TO GET STARTED...

https://mickwebsite.com/CrawlHelps/AboutCrawlNScrape.html

Mick

[email protected]