Amazon not only hurts small stores when it steals their customers, but it also hurts small stores by disrupting their websites. Amazon and businesses that host with Amazon (on AWS) send bots to crawl competitors' websites. These bots take up bandwidth and create errors that delay real users from accessing the sites.
In the example shown, we see Amazon or one of its clients using AWS to crawl the indie store Dalton Brody and create an error that only a bot would create.
We did not ask Amazon or our client's competitors that are hosting with Amazon to crawl our sites.
How do we stop this? There is no easy way. One may suggest blocking an IP address, yet just today, we've had four different Amazon IP addresses crawling our clients' pages. In other words, we block one IP and more crawl us. If we were to block all of Amazon's IPs, we'd not receive legit emails from addresses that are hosted on Amazon's AWS. We have clients that do use Amazon to host their email, so we can't simply block all of Amazon. Maybe Amazon mixes its email and bot IPs so businesses like us can't apply a blanket block to them.
Here is the Amazon IP the crawled our indie store client:
View Post at Bridge
hurting indie stores