Understanding the architecture, content, internal linking and other onpage factors of your website becomes more and more important. As such, the reliance on spider tools to crawl and optimize websites is rather unsurprising. The largest problem here is the crawl depth, quality of findings, display of results
Here are 7 of the best on-page crawlers (in no particular order) currently available to webmasters looking for an edge:
Screaming Frog
The Screaming Frog SEO Spider Tool is a comprehensive crawler available for Windows, Mac OS X and Ubuntu Linux. First things first, it’ll give you a complete rundown of all the pertinent information pertaining to a domain such as outgoing links, incoming links, meta tags and keywords, titles, images and response codes. Furthermore, it allows users to export their data to CSV files for further dissemination. The free version will only index 500 links per domain, so you’ll need the paid version for big projects.
Xenu
A longstanding favorite of SEOs around the world, Xenu’s Link Sleuth is the go-to spider tool for many. Though a bit long in the tooth at this point, it’s still a robust program with a lot to offer. It provides basic link intelligence for any domain and doesn’t cost a penny. Originally designed to check domains for broken links, it’s a spartan yet effective link crawler. It works on SSL domains, can scan Java applets and will spit out reports on redirects of all kinds quickly.
OnPage
Currently only available in German, OnPage is nevertheless a powerful on-page crawler regardless of your native tongue. So much more than a mere link spider, it’ll give you loads of information on any domain including ranking keywords, ranking pages, click-through rates, meta tags and links from the “bad neighborhoods” of the web. Of course, it also gives you alerts on broken links and re-directs. Available online, OnPage can also export your raw data to either the CSV or Excel file formats. Overall…extremely powerful tool, that saves you tons of time!
DeepCrawl
The self-proclaimed “ultimate site auditor” known as DeepCrawl is a combination link inspector and analytics powerhouse that covers most if not all of the SEO bases. It goes beyond simple link indexing and digs deeply into the internal structure of a domain to give the user a thorough dossier on a site’s SEO health. In particular, DeepCrawl places an emphasis on rooting out potential pitfalls like duplicate content that factor heavily into Google’s current ranking algorithms. If you need serious SEO intelligence, DeepCrawl’s a safe bet.
80legs
Already known by many search experts, 80legs is a page crawler built for today’s highly social web. The primary selling points of 80legs are its detailed, pre-compiled “crawl packages” for popular social networking sites, real estate directories, consumer review portals and more. Basically, 80legs is all about providing link reports on sites with a lot of data that changes frequently. They offer serious, enterprise-grade web analytics services to SEOs who work with major corporate clients. Though 80legs is by no means cheap, it’s one of the best crawlers around.
Searchmetrics
One of the most versatile SEO tool kits you can get your hands on nowadays, the Searchmetrics Suite will tell you everything you need to know about a domain’s link structure and then some. Searchmetric’s highly customizable dashboard puts you in the driver’s seat when you’re analyzing pages for areas that could use improvement. In addition to serving up detailed synopses of inbound and outbound links, it allows you to filter out irrelevant data as you see fit and even compare your pages to benchmark domains to see how you’re doing. Their new SEO Optimizer was a valuable addition to anyone doing onpage audits.
SISTRIX Smart
Like OnPage, SISTRIX Smart is a German crawler program that delivers extremely accurate and timely reports on any domain. Absolutely free to use, it allows SEOs to quickly visualize traffic, link structures and negative aspects of a site at a glance. If you’re performing basic link cleanup on a site, it will give you pointers on the most effective ways to optimize pages for the best SERPs results possible. Though not the most full-featured crawler found online, it’s one of the best free tools on the market.
Use What Works, Lose the Rest
Just as it’s important to stay current on the latest trends in search, it’s also vital that you stay up-to-date on the best crawlers to use in pursuit of your goals. While some programs are renowned for their longevity, many quickly fade away due to lack of interest or innovation. As such, it’s important to regularly evaluate the effectiveness of the tools you rely on for SEO work to ensure that you’re getting the most from your efforts.
Do you know of any other great onpage crawler! Contact me!
