"Bluesky scraper" gets searched by two very different people: someone who wants a backup of their own account, and someone building a research dataset or monitoring tool. Bluesky handles both better than any comparable platform, because the AT Protocol was designed for open data access rather than defending against it.

Here is how to do each properly.

Why Bluesky Is Different

On most social platforms, scraping means fighting the platform – rotating proxies, parsing HTML, evading detection. On Bluesky, none of that applies.

The AT Protocol publishes user data as signed public records replicated across the network and streamed over a public firehose. Reading it is the intended behavior. There is no adversarial relationship, no anti-bot arms race, and no need for a headless browser.

This changes the engineering problem entirely: it is a data volume and rate limit problem, not an evasion problem.

Case 1: Exporting Your Own Account

If you want your own posts, this is straightforward and you should do it regardless.

The CAR file export

Your entire repository can be exported as a CAR file – a content-addressed archive containing every record your account has created: posts, likes, follows, blocks, lists, profile history. It is complete and it is portable, which is the whole point of the protocol's design.

The catch: a CAR file is not human-readable. You need a tool to convert it into something you can actually browse.

Converting it to something useful

Backup tools handle the conversion into readable HTML and structured JSON. Our own Bluesky account backup tool does this, and we walk through the process in how to back up a Bluesky account.

What is not in the export

Important gaps to know about:

  • Direct messages and group chats. These are not AT Protocol records – they run on separate infrastructure and are not in your repository. See our group chats post.
  • Images and video blobs. Media is stored separately from records; a repository export references blobs rather than embedding them. A complete archive requires fetching those too.
  • Other people's replies to you. Those live in their repositories, not yours.

Case 2: Collecting Network-Wide Data

If you are building a research dataset, an analytics product or a monitoring tool, you have two paths and the choice matters a lot.

Path A: streaming (almost always correct)

Consume the firehose or Jetstream and capture events as they happen. This gives you everything in real time, does not consume request rate limits, and scales to the full network.

Use Jetstream unless you specifically need cryptographic verification – it delivers plain JSON at roughly a tenth the bandwidth and near-zero CPU, with server-side filtering by collection and DID. Our firehose vs Jetstream comparison covers the decision in detail.

The limitation: streaming only gives you data from the moment you connect. It is not a time machine.

Path B: backfilling via the API

For historical data, you query endpoints directly – author feeds, follower graphs, list membership, individual records. This works and it is documented, but it is where people get into trouble.

The pitfalls:

  • Rate limits are real. Enumerating followers for large accounts or crawling millions of profiles will hit them fast. Read the ratelimit-remaining header and throttle proactively rather than reacting to 429s – see our rate limit guide.
  • Pagination is cursor-based. Persist cursors durably; do not restart from scratch on failure.
  • Scale is genuinely large. Roughly 46 million registered accounts. Crawling all of them is a serious infrastructure commitment, not a weekend script.
  • Cache aggressively. DID documents, handle resolution and profile data change rarely.

The hybrid that actually works

Most production systems do both: stream forward, backfill selectively. Connect to Jetstream immediately so you stop losing data, then backfill history only for the specific accounts or topics you need. Trying to backfill the entire network before you start streaming is the classic mistake.

Where the Lines Are

Data being public does not mean every use of it is appropriate. The protocol makes collection easy; it does not make judgment unnecessary.

Generally fine

  • Backing up your own account
  • Aggregate analytics – trends, volume, hashtag popularity
  • Building feeds, search and discovery tools
  • Brand and keyword monitoring on public posts
  • Academic research with appropriate ethical review

Where it gets problematic

  • Republishing deleted content. When someone deletes a post, the protocol emits a deletion event. Honor it. Your archive continuing to serve something the author removed is the single most common way well-intentioned projects cause harm.
  • Building profiles on individuals. Aggregating one person's entire activity into a dossier is a materially different act from aggregate analysis, even when every input was public.
  • Republishing at scale without consent. "It was public" is not a complete answer to "you built a searchable database of everything I have ever said."
  • Training data. A live controversy across the platform, and many users migrated to Bluesky specifically over this issue. Be aware of what you are wading into.
  • Ignoring account deletion. If a repository is tombstoned, your copy should reflect that.

The practical standard we would apply: honor deletions, aggregate rather than individuate, and be able to explain your dataset to the people in it without embarrassment. Privacy law – GDPR in particular – applies to public data too, and "the API let me" is not a legal defense.

Should You Build This At All?

Often, no. Running a stream consumer means operating infrastructure continuously to answer a question someone already answers.

If you need trending hashtags, we expose them as an API. If you need keyword monitoring, Keyword Alerts runs the pipeline for you. Build your own when the data pipeline is the product, not when it is a dependency.

Frequently Asked Questions

Do I need an API key to read Bluesky data?

Public reads and the firehose do not require authentication. Anything acting on behalf of an account does – see our OAuth guide.

Can I scrape another user's posts?

Their public posts are publicly readable by design. What you then do with them is where the responsibility sits.

How do I get historical Bluesky data?

Query author feeds and repository records directly, or use a published research dataset. The firehose is forward-looking only.

Will I get banned for scraping?

Reading public data within rate limits is normal use. Ignoring rate limits, hammering endpoints or building something that abuses users is a different matter.

Tools That Already Do This

Download Skyscraper for iOS →

Related Reading