The Bot Handbook: Scraper bot

A scraper is a bot that collects web application content and other data for use elsewhere. It crawls the pages of a website, accessing the available data and copying the content. The stolen content is then published elsewhere, or used alternatively depending on the attacker’s motives.

This is a hard-to-detect activity commonly used in eCommerce by competitors to monitor prices (especially in a dynamic pricing environment), product availability, ratings, and inventory. The presence of these bots on a website can lead to search engine ranking degradation, which can be devastating to a business that relies heavily on web searches.

Search engines recognize stolen content and label the duplicate web page as plagiarized, with corresponding ranking implications. In particularly extreme cases, the search engine may deem the copy to be the original source of content, and there are various ways an attacker can achieve this if they are determined.