The 7 Best On-Page Crawlers

Understanding the architecture, content, internal linking and other onpage factors of your website becomes more and more important.  As such, the reliance on spider tools to crawl and optimize websites is rather unsurprising. The largest problem here is the crawl depth, quality of findings, display of results Here are 7 of the best on-page crawlers (in no particular order) currently available to webmasters looking for an edge:

Screaming Frog The Screaming Frog SEO Spider Tool is a comprehensive crawler available for Windows, Mac OS X and Ubuntu Linux. First things first, it'll give you a complete rundown of all the pertinent information pertaining to a domain such as outgoing links, incoming links, meta tags and keywords, titles, images and response codes. Furthermore, it allows users to export their data to CSV files for further dissemination. The free version will only index 500 links per domain, so you'll need the paid version for big projects.

Xenu A longstanding favorite of SEOs around the world, Xenu's Link Sleuth is the go-to spider tool for many. Though a bit long in the tooth at this point, it's still a robust program with a lot to offer. It provides basic link intelligence for any domain and doesn't cost a penny. Originally designed to check domains for broken links, it's a spartan yet effective link crawler. It works on SSL domains, can scan Java applets and will spit out reports on redirects of all kinds quickly.

OnPage Currently only available in German, OnPage is nevertheless a powerful on-page crawler regardless of your native tongue. So much more than a mere link spider, it'll give you loads of information on any domain including ranking keywords, ranking pages, click-through rates, meta tags and links from the "bad neighborhoods" of the web. Of course, it also gives you alerts on broken links and re-directs. Available online, OnPage can also export your raw data to either the CSV or Excel file formats. Overall...extremely powerful tool, that saves you tons of time!


DeepCrawl The self-proclaimed "ultimate site auditor" known as DeepCrawl is a combination link inspector and analytics powerhouse that covers most if not all of the SEO bases. It goes beyond simple link indexing and digs deeply into the internal structure of a domain to give the user a thorough dossier on a site's SEO health. In particular, DeepCrawl places an emphasis on rooting out potential pitfalls like duplicate content that factor heavily into Google's current ranking algorithms. If you need serious SEO intelligence, DeepCrawl's a safe bet.

80legs Already known by many search experts, 80legs is a page crawler built for today's highly social web. The primary selling points of 80legs are its detailed, pre-compiled "crawl packages" for popular social networking sites, real estate directories, consumer review portals and more. Basically, 80legs is all about providing link reports on sites with a lot of data that changes frequently. They offer serious, enterprise-grade web analytics services to SEOs who work with major corporate clients. Though 80legs is by no means cheap, it's one of the best crawlers around.

Searchmetrics One of the most versatile SEO tool kits you can get your hands on nowadays, the Searchmetrics Suite will tell you everything you need to know about a domain's link structure and then some. Searchmetric's highly customizable dashboard puts you in the driver's seat when you're analyzing pages for areas that could use improvement. In addition to serving up detailed synopses of inbound and outbound links, it allows you to filter out irrelevant data as you see fit and even compare your pages to benchmark domains to see how you're doing. Their new SEO Optimizer was a valuable addition to anyone doing onpage audits.

SISTRIX Smart Like OnPage, SISTRIX Smart is a German crawler program that delivers extremely accurate and timely reports on any domain. Absolutely free to use, it allows SEOs to quickly visualize traffic, link structures and negative aspects of a site at a glance. If you're performing basic link cleanup on a site, it will give you pointers on the most effective ways to optimize pages for the best SERPs results possible. Though not the most full-featured crawler found online, it's one of the best free tools on the market.

Use What Works, Lose the Rest Just as it's important to stay current on the latest trends in search, it's also vital that you stay up-to-date on the best crawlers to use in pursuit of your goals. While some programs are renowned for their longevity, many quickly fade away due to lack of interest or innovation. As such, it's important to regularly evaluate the effectiveness of the tools you rely on for SEO work to ensure that you're getting the most from your efforts.

Do you know of any other great onpage crawler! Contact me!

Social Graph Gaming

I usually don't post about SEO anymore, but if I hear all this paid, owned, earned social stuff out there I just feel like I have to defend SEO and especially the value of links a bit. SEO is currently and will be in the near future the online marketing techniques that drives the highest ROI (if done properly). Social engagement or whatever you want to call it is still in the early days and until Facebook, Google Plus and Co. are gamed like below...I just don't see a chance that the simplicity of the "link system" will be taken over by a social graph.

Nowadays it is way easier and especially much cheaper to acquire social likes, shares, tweets from real users than getting  a valuable link from an authoritative website! The example  below is just one sample of how you can game the social graph. There are plenty of other "tricks", such as contests or third party software products like,, or

Long story short (No...I don't write a 20 page post to pretent thought leadership): We are far far away from having the social graph taking over the SERPs in Google & Co.. Social is an imoportant part of  marketing, but it is not a replacement and rather a supplement to SEO.  Especially at a time where Google Plus, Twitter and Facebook accounts can be sold, traded and gamed way worse than anyone ever could imagined.


updated:  2/6/2012 19:38



Visibility Indexes And The Super Long-Tail

In 2011 the major headlines of the search publications covered the Panda update and Google’s war on the long-tail. Within the search industry, but also in larger publications it has been widely reported that Google’s Panda update was introduced to eliminate rankings for sites of little value. Companies such as Demand Media have been literally trashed across the web for polluting the web with low quality content.

Influential bloggers and even main stream media called “Panda” the end of content farms like, the of price comparison portals such as Pronto, the death of review sites such as and ultimately a great thing for the internet.

This data was all backed up by SEO Visibility scores from various vendors across the globe. But what did really happen:

The Panda update hit only a tiny fraction of the search queries that are daily going on. This has to do with the fact that up to 25% of the daily searches are complete new queries. This could be new words, trends, inventions, brands or other stuff that just nobody typed in before.

That’s why everyone in the industry needs to make sure to remember that visibility scoring is based on a limited amount of data. It’s a great indicator about where the wind might be blowing at that particular moment in time, but it’s far from being accurate.

The issues with accuracies just has to do with the fact that the SEO Software market is very fragmented and larger companies, which would allow significant infrastructure investment, do not exist in the SEO industry. Even the leaders in SEO enterprise software have active revenues under $15 million / year.

If you take all this in consideration it is not surprising why companies like Demand Media still print a good junk of money despite their visibility score being down.

Even certain industries such as e-commerce shops with lots of products such as Custom Printing by Print Lion or massive content download sites such as Tradebit are still doing very well b/c their super long-tail traffic hasn’t been affected.

Interview with Heiner Hemken

Thanks for stopping by and giving us an interview for the Webanalyticsbook readers.For people, who don’t know him, Gretus runs, one of the top search engine optimization blogs in Germany. Heiner is currently also working on a secret SEO Tool, which he will roll out within a few months.

Webanalyticsbook: Since we already spoke a few times, let’s start the interesting stuff right away. Why another SEO Tool?

Gretus: In Germany there are only a few tools, which offer all the relevant information for webmasters. There are a bunch of great tools, but most are quite expensive. We want to offer simple tools, which can be used by everyone and most probably will be free.

Webanalyticsbook: What functionalities do you want to include in your tool?

Gretus: As already said, our tool will only deliver the basic information, e.g. keyword positions, number and quality of backlinks, PageRank, TrafficRank and so on.

Webanalyticsbook: How does this differ from tools such as Sistrix, Searchmetrics or SEOlytics, which are the three big one’s in Germany?

Gretus: You can´t compare our tool to them. The named tools are more or less huge instruments, specially made for companies and SEOs. If you´re a webmaster who only earns a few dollars with a site, you can´t affort a tool which costs hundreds of dollars a month.

Webanalyticsbook: The tool will be completely free?

Gretus: It will be completely free!

Webanalyticsbook: Do you plan to offer an API for more advanced clients or will you offer a closed system like the most vendors?

Gretus: First we will start out with a closed system, too. We are planning a kind of networking component, by which all the users can change the tool and build it out.

Webanalyticsbook: How much further do you want to develop the tool? Will it be a fully-equipped tool like Searchmetrics or rather stand in the middle?

Gretus: As you know from our project, everything is possible. If the users like our tool, we’ll build it out

Webanalyticsbook: Thanks for the interview! I’ll follow you on Twitter and good luck with your soon-to-be launched SEO tool.

14 year-old SEO Superstar

After speaking at SEO-Campixx in Berlin a few weeks ago, I was quite busy and totally forgot to mention Justin, a 14-years old SEO and Online marketer. He is not only blogging since he was 12,  but also has an exhaustive knowledge about SEO, Online marketing.
Since this is a respectable accomplishment and b/c he promised me not to drop out of school to become a full-time SEO, here the promised link.Also please keep spying on other SEOs in the Captain's Bar :)

Pic credit to Gerald

SEO Monitoring

Andre Scholten, a Dutch SEO, wrote a great post (in Dutch) about SEO tracking / SEO Analytics. It's an interesting idea to track SEO efforts with Google Analytics and the way it can be setup, is done really smart. Yoast, one of the most famous Wordpress-SEO gurus, allowed Andre to re-post it on his blog (in english) and it draw quite some interest.

The post is very detailed and written very well, however I miss the part where some warnings should have been posted. If a user installs Google Analytics, they have to be aware of what they are getting into. I am a big fan of G Analytics and I don't want to go in details, but some sites should stay away of tracking their linkbuilding / SEO efforts with a tool from a company that also runs a search engine, a  toolbar, an ad network, an email business ... .

It's not about being paranoid, but there should also be caution when it comes to free tools, even if they have a big Google logo on top of it.

Same also is the case for other G tools like Adwords. I for myself will never run an Adwords campaign on a not 100% ready site again.  And I know why :)

How to Increase International Google Rankings By Up To 30% it's finally official: After a comprehensive study, NTT Europe online published that local Google Rankings depend tremendously on where you host.

I've noticed that with a few sites in France, Germany and Australia over the past 2-3 years. Without hosting these sites locally, it's quite difficult to rank for anything.

UK SERPs however weren't effected as dramatically as continental Europe. So rule of thumb:

1. Get a local TLD ( /.de/.fr...)
2. Host locally (double check if servers are really located in the country. Especially European hoster often  run their servers in Germany or even in the US).
3. Build /buy inbound links from "local" sites
4. Register your domain in local directories as well as DMOZ.
5. Enjoy better rankings

Found via Seodeluxe

How To Drive International Traffic Without Speaking the Language

When I was looking for "Webanalyse" (Webanalytics in German) in the SERP's of I found an interesting result:

Avinash's WebAnalytics Blog translated in German:

Since I've met Avinash a few times and I kind of did not expect him to speak German I was quite surprised about  his site in the SERP's.

First I was sure that he might have  installed Google Language Tools (since he works for Google and the translation wasn't perfect). However it did not make sense since all the language URLs are extremely clean and pretty much optimized for the engines.

His site also has the extensions of  for French or for Spanish...and probably a few others. thought: It's a wordpress plugin. And I believe..this is it:
Taragana Plugin Pro

The plugin is quite nice programmed since it creates clean URLs, caches pages and uses 4 translation engines. It also supports 32 languages and most importantly Google seems to like it.

So if you need more content or you are trying to engage with an international audience, this is a great way to drive traffic.

How much is your No.1 ranking in Google worth

I've previously written about my basic approach of figuring out how much a No.1 ranking in Google is worth. Aaron over at Seobook went probably a little overboard when he put his 15 page(!) "How much is a Top Google Ranking worth" posting together. However I think this is one of the best SEO postings out there, but you have to be ready to put massive efforts into it by using all his data sources and techniques.

Overall if you are serious about SEO, his approach is really cool  even he forgot to include the Wiki tracker, which in my eyes gives great accuracy on volume of traffic on a specific keyword.

Just because of the simple reason that Wikipedia ranks mostly in the top 3 for big keywords.