AnalysisBot

A small, polite crawler that reads public product prices from online shops in Bangladesh.

Who we are

AnalysisBot is run by a retail business in Bangladesh. We use it to compare the public prices of products we also sell, so that we can price our own products fairly. The results are for our internal use only. We do not republish your content, product descriptions or images.

How to recognise it

Every request carries this User-Agent header:

AnalysisBot/1.0 (+https://analysis-bot-info.pages.dev/bot)

We never change or hide this identity, and we never pretend to be a browser.

What it reads

• Your robots.txt and your sitemaps.
• Product pages, and the public product data your platform already publishes (for example a Shopify or WooCommerce product feed).
• From those pages: product name, brand, price, regular price, stock status, barcode or SKU, and the product URL.

It never downloads images, CSS or JavaScript files. It never logs in, submits forms, adds to cart or checks out. It collects no personal data.

How it behaves

• It follows the Allow and Disallow rules in your robots.txt for AnalysisBot (or for * when there is no AnalysisBot group). It re-reads robots.txt at least once every 24 hours. If robots.txt answers with a server error or 429, or cannot be reached, it fetches nothing in that run.
• At most one request per second to your site, and slower if your robots.txt sets a Crawl-delay.
• It only follows URLs from your sitemaps, your public product data or your listing pages. It does not guess URLs.
• It sends conditional requests (If-None-Match, If-Modified-Since) where it can, so unchanged pages cost you little.
• On 429 or 503 it waits for your Retry-After (60 seconds if none is given) and retries at most twice. If you ask for a wait longer than 10 minutes, it stops at once and does not come back until the time you asked for (at most 7 days; at least 1 hour).
• On 403, a challenge page or a captcha, it stops visiting your site. A person on our side then reviews it; it does not retry on its own.
• It does not use proxies or rotate IP addresses to get around limits or blocks.
• Larger reads run mostly at night, Bangladesh time (01:00 to 06:00), with lighter checks during the day.

How to block it

Add this to your robots.txt. It is picked up within 24 hours:

User-agent: AnalysisBot
Disallow: /

You can also block only part of your site with a narrower Disallow rule, or slow it down with Crawl-delay, for example:

User-agent: AnalysisBot
Crawl-delay: 5

If your site asks us to wait with 429 or 503 and Retry-After, we wait. If it blocks us, we stop and do not try to get around the block.