- Deepcrawl’s enterprise-scale SaaS platform for technical SEO and website intelligence leverages best-in-class serverless architecture design in its latest crawler to drastically enhance website crawling speeds
- Roll-out of the enhanced crawler is underway to new and existing Deepcrawl users
Deepcrawl, an enterprise-scale website intelligence and technical SEO platform, has launched a new, enhanced version of its market-leading website crawler. The latest crawler developments will introduce ultra-fast crawl speeds to the Deepcrawl platform, breaking earlier records with speeds of up to 450 URLs per second for non-rendered pages and 350 URLs per second for JavaScript-rendered pages. These crawl speeds were reached in field testing on enterprise-scale websites and offer a major time-saving boon for teams working on large sites that frequently house hundreds of thousands, if not millions, of pages.
The enhanced crawler allows for unprecedented scalability on a platform that already serves many of the world’s largest brands, opening the door to new efficiencies in website analysis. Roll-out of the new crawler has begun, with all users expected to have access to the enhanced platform by September 2022.
Crawling software is a foundational aspect of SEO and website intelligence platforms. It traverses a website’s pages to collate the raw data required for sophisticated website analytics and serves as the first step in understanding and optimizing a website’s technical health and search performance. Earlier crawling technology could take several days to finish crawling an enterprise-size website — by comparison, Deepcrawl’s enhanced crawler can crawl a million pages in the span of just one hour.
Increased flexibility in data collection is also built into this latest crawler release. The information that can be captured in website crawls has been expanded beyond traditional technical SEO metrics, paving the way for new built-in metrics and reporting options within the Deepcrawl platform. Presently, users have access to 250+ built-in reports as well as the ability to design custom extractions that collect data on any specified website element. The added flexibility in data collection allows for new use cases to be developed via Deepcrawl’s crawler, with a recently released GraphQL API providing easy implementation for developers.
“Deepcrawl’s latest advancements in crawling technology allow for tailor-made website intelligence at unparalleled speeds. The enhanced crawler makes it easier than ever for businesses to make effective decisions about their online presence,” said Ali Habibzadeh, Chief Technology Officer at Deepcrawl.
Tune in to Martech Cube Podcast for visionary Martech Trends, Martech News, and quick updates by business experts and leaders!