emojistats backfill

crawling every Bluesky repo to rebuild emoji history since 2023

crawling · updated 2s ago3:52:17 PM
shard1 snapshot · 2.2dshard2 snapshot · 2.2dshard3 snapshot · 20sshard0 snapshot · 17sshard5 snapshot · 4sshard4 snapshot · 2s

2 shard snapshots are stale — counts below use each shard's latest telemetry row

Crawl progress
95,440,578 of 95,470,241 enumerated repos resolved

100.0%

posts loaded

2.6B

2,623,769,667

data downloaded

7.8 TiB

repos/min

16

rolling 10 min

rows/s

0

into ClickHouse

ETA

30.4h

pending + fetching

in flight

41

6 shards

30,360,868 unreachable parked (retry waves + final sweep) — outside the ETA

verification
loose-round-20260614T2140Z · finished
loaded check

14,009,955 / 14,009,955 repos classified across latest reporting shard runs · 100.0%

exact

13,325,082

loose

684,873

loaded open

37

historic diff

0

loose file

684,873

reporting shards

6 / 6

37 post-recrawl repos remain loaded pending loaded-only verification

recheck: whale-2026

updated 1.7d ago

loose recrawl
fail-loose-recrawl-20260614T2140Z · idle
seen

655,944 / 684,873 repos completed · 95.8%

active shards

0 / 4

in flight

0

loaded repos

654,718

issue repos

1,166

event rows

690,643

repos/min

0

eta

—

post rows/sec

1,543

updated 14.9h ago

v2 metadata recrawl
prepared; waiting for current crawl + final sweep
ready

target repos

3,208,370

target posts

478,676,984

current crawl left

29,663

recrawl eta

not started

Emojis of the recovered past
top 10 across 789M backfilled posts with emojis

🟩

71.3M

😂

68.2M

😭

66.3M

🤣

52.3M

❤️

38.3M

✨

37.2M

⬜️

34.6M

🔥

28.5M

🥰

26.3M

😍

24.6M

Oldest post recovered
how deep into history the crawl has reached

2022-11-16

3.6y ago · around the first Bluesky sandbox posts

Filling in history
4.5B surviving posts recovered so far, by month written — the bars grow as the crawl reaches deeper into the past
Posts throughput
project lifetime · posts loaded per minute and per second
Download rate
project lifetime · repo archive data fetched per minute
Repo status breakdown
loaded is every fetched repo with post rows; verified is the subset whose post rows have passed the digest check. residual pending/fetching here is retained canonical telemetry residue from retired/stale shard snapshots, not active backfill work.

pending

29,659

fetching

4

loaded

14,009,933

verified

10,109,270

empty

26,396,607

tombstoned

2,305

deactivated

423,822

takendown

2,911,622

unreachable

30,360,868

quarantined

17,095

failed

21,318,326

Repo issue reasons
latest 6-shard ledger rollup · grouped from SQLite error text

unreachable

30,361,263

bulk parked dead host (generic)22,648,155
pds.trump.com dead/DNS7,218,576
other host-dead parked319,071
other/unclassified150,129
http 429/rate limited10,093
referendumapp 429/502 tail9,015
dns ENOTFOUND6,218
timeout/stall6

quarantined

16,948

malformed CAR: decoded remainder16,722
malformed CAR: invalid arg encoding102
malformed CAR: EOF varint52
malformed CAR: EOF data21
missing MST node block19
post record missing from CAR13
expected MST node block11
expected commit block6
missing commit block1
invalid binary CID1

failed

21,318,072

PLC-only DID missing from host listRepos19,782,410
getRepo RepoNotFound1,410,158
http 40470,842
other http 4xx54,639
other/unclassified23
Top PDS hosts
by terminal repos seen
hosttotalloadedemptyissues

jellybaby.us-east.host.bsky.network

18.6 GiB · avg 48 posts

1M146K712K207K

stropharia.us-west.host.bsky.network

24.1 GiB · avg 136 posts

1M145K710K180K

morel.us-east.host.bsky.network

300.8 GiB · avg 378 posts

738K297K172K269K

hebeloma.us-west.host.bsky.network

26.4 GiB · avg 96 posts

631K94K413K125K

bankera.us-west.host.bsky.network

26.9 GiB · avg 92 posts

614K93K412K109K

entoloma.us-west.host.bsky.network

27.3 GiB · avg 102 posts

594K93K412K88K

calocybe.us-west.host.bsky.network

26.9 GiB · avg 97 posts

580K93K414K73K

gomphidius.us-west.host.bsky.network

25.4 GiB · avg 88 posts

578K93K412K72K

hygrophorus.us-west.host.bsky.network

25.6 GiB · avg 92 posts

573K93K412K69K

leccinum.us-west.host.bsky.network

27.4 GiB · avg 106 posts

561K94K413K55K

pholiota.us-west.host.bsky.network

27.1 GiB · avg 95 posts

561K93K413K55K

cortinarius.us-west.host.bsky.network

29.6 GiB · avg 90 posts

545K93K414K38K
Recent issues
failed, quarantined, unreachable, takendown, deactivated
  • failed072004.xyz9m ago

    did:plc:q6hnuqslvuyzt22azebcqgmj

    getRepo did:plc:q6hnuqslvuyzt22azebcqgmj@072004.xyz: http 404

  • failedexample.com9m ago

    did:plc:qnseipta2yd3xsp3qybmnyjg

    getRepo did:plc:qnseipta2yd3xsp3qybmnyjg@example.com: http 404 <!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,">

  • failedkt.tngl.oyster.cafe9m ago

    did:plc:srgecn2oqr5mxcgzl7r43gjy

    getRepo did:plc:srgecn2oqr5mxcgzl7r43gjy@kt.tngl.oyster.cafe: http 404 404 page not found

  • failedknot1.tangled.sh10m ago

    did:plc:44b2ltw35lrcycn675wonxse

    getRepo did:plc:44b2ltw35lrcycn675wonxse@knot1.tangled.sh: http 404 404 page not found

  • failedexample.com10m ago

    did:plc:hhmftfcjmqa7otdeyfnhx4lb

    getRepo did:plc:hhmftfcjmqa7otdeyfnhx4lb@example.com: http 404 <!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,">

  • failedexample.com10m ago

    did:plc:ksufvgpw3twzy2bkyvrg2jiw

    getRepo did:plc:ksufvgpw3twzy2bkyvrg2jiw@example.com: http 404 <!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,">

  • failedkt.tngl.oyster.cafe10m ago

    did:plc:34wn6qq3vwil5j42ourzbu6s

    getRepo did:plc:34wn6qq3vwil5j42ourzbu6s@kt.tngl.oyster.cafe: http 404 404 page not found

  • failedkt.tngl.oyster.cafe10m ago

    did:plc:4hft2kiuvznt6aolvefhvo43

    getRepo did:plc:4hft2kiuvznt6aolvefhvo43@kt.tngl.oyster.cafe: http 404 404 page not found

  • failedknot1.tangled.sh10m ago

    did:plc:3ul3xphk4wznx3p7omql5c2y

    getRepo did:plc:3ul3xphk4wznx3p7omql5c2y@knot1.tangled.sh: http 404 404 page not found

  • failedknot1.tangled.sh10m ago

    did:plc:3vpvatckte43oouirtfzeeup

    getRepo did:plc:3vpvatckte43oouirtfzeeup@knot1.tangled.sh: http 404 404 page not found

  • failedknot.candy123.moe10m ago

    did:plc:fr3ulrmij4kx6mwtqjlkdyit

    getRepo did:plc:fr3ulrmij4kx6mwtqjlkdyit@knot.candy123.moe: http 404

  • failedknot1.tangled.sh10m ago

    did:plc:44pj7fslwnnm7mvtqtwoxyi2

    getRepo did:plc:44pj7fslwnnm7mvtqtwoxyi2@knot1.tangled.sh: http 404 404 page not found

  • failedknot1.tangled.sh10m ago

    did:plc:6r3fwo2mkuo3tmlr4piyo774

    getRepo did:plc:6r3fwo2mkuo3tmlr4piyo774@knot1.tangled.sh: http 404 404 page not found

  • failednercone.dev10m ago

    did:plc:5ipa4ihy4ccmtfhu5g2hp4c5

    getRepo did:plc:5ipa4ihy4ccmtfhu5g2hp4c5@nercone.dev: http 404 <!-- For Bots: On this site, you can read any page in Markdown by using cURL or by replacing

  • failedknot1.tangled.sh10m ago

    did:plc:43pcwdqjh3cyhq3vg2u3cf6x

    getRepo did:plc:43pcwdqjh3cyhq3vg2u3cf6x@knot1.tangled.sh: http 404 404 page not found

  • failedpoisonpie.us-west.host.bsky.network10m ago

    did:plc:vtkpjm6syfs25bunv3bfcbnn

    getRepo did:plc:vtkpjm6syfs25bunv3bfcbnn@poisonpie.us-west.host.bsky.network: http 400 NotFound {"error":"NotFound","message":"Repo not found"}

  • failedexample.com10m ago

    did:plc:ztoaflrhv3khclocx3rd5gdy

    getRepo did:plc:ztoaflrhv3khclocx3rd5gdy@example.com: http 404 <!doctype html><html lang="en"><head><title>Example Domain</title><link rel="icon" href="data:,">

  • failedknot.candy123.moe10m ago

    did:plc:akjog4j273ktslc4tvo2ca4s

    getRepo did:plc:akjog4j273ktslc4tvo2ca4s@knot.candy123.moe: http 404

  • failedknot.candy123.moe10m ago

    did:plc:2arb5qsdqtl43k6zxcrujgho

    getRepo did:plc:2arb5qsdqtl43k6zxcrujgho@knot.candy123.moe: http 404

  • failedknot1.tangled.sh10m ago

    did:plc:3i5k7wduamocjr6btruerrx6

    getRepo did:plc:3i5k7wduamocjr6btruerrx6@knot1.tangled.sh: http 404 404 page not found