⚛ AttentionHub
A live study · running on this hub

We re-ran the code behind 173 papers.

Not "does the PDF look right" — we built each repo, ran the experiment, and checked the headline number against what the paper claimed, with a tolerance fixed in advance. Here's what came out.

173 attempted · updating as the batch runs
73%
reproduced their headline result as shipped
127 of 173 papers, within a pre-registered tolerance
84%
even built & ran at all
the rest failed to build or crashed on launch
87%
matched, given the code produced a number
so even when it runs, the number often disagrees

Every outcome, honestly

One bar, 173 papers. Failures aren't noise — they're the finding.

127
19
127 reproduced as shipped 19 ran, but the number didn't match 9 built, but the experiment crashed 11 environment wouldn't build 7 exceeded the time budget

Where papers drop off

Most of the loss happens before a number is ever produced — an environment/packaging gap, not a wrong result.

attempted
173 · 100%
environment built
162 · 94%
ran to completion
146 · 84%
produced a number
146 · 84%
matched the claim
127 · 73%

Packaging predicts reproduction

Papers shipped with a clean, pinned environment reproduce far more often than a bare repo — direct evidence for machine-actionable packaging.

pip_env
109/128 · 85%
repo_artifact
18/39 · 46%
pip
0/6 · 0%

Every paper, one tile

173 papers, 173 tiles — colored by what happened. Hover for the paper; the hub owns the assertion, so authors can't self-pass.

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠
✓ reproduced ≠ ran, off / no number ✗ wouldn't build / crashed / timed out
Full table claimed vs got, per paper
paperoutcomeclaimedgoterr
anomaly-ecod-internetads-auc REPRODUCED 0.676 0.677 0.148%
anomaly-ecod-spambase-auc REPRODUCED 0.701 0.6556 6.476%
anomaly-ecod-speech-auc REPRODUCED 0.485 0.4697 3.155%
anomaly-ecod-stamps-auc REPRODUCED 0.939 0.8761 6.699%
anomaly-ecod-wdbc-auc REPRODUCED 0.983 0.9706 1.261%
anomaly-hbos-breastw-auc REPRODUCED 0.983 0.9851 0.214%
anomaly-hbos-cardio-auc REPRODUCED 0.835 0.8377 0.323%
anomaly-hbos-hepatitis-auc REPRODUCED 0.796 0.7887 0.917%
anomaly-hbos-internetads-auc REPRODUCED 0.699 0.683 2.289%
anomaly-hbos-lympho-auc REPRODUCED 0.996 0.9977 0.171%
anomaly-hbos-mammography-auc REPRODUCED 0.834 0.8299 0.492%
anomaly-hbos-optdigits-auc REPRODUCED 0.873 0.8702 0.321%
anomaly-hbos-pima-auc REPRODUCED 0.659 0.6858 4.067%
anomaly-hbos-satellite-auc REPRODUCED 0.758 0.7659 1.042%
anomaly-hbos-satimage2-auc REPRODUCED 0.98 0.9716 0.857%
anomaly-hbos-spambase-auc REPRODUCED 0.67 0.6627 1.09%
anomaly-hbos-speech-auc REPRODUCED 0.454 0.4757 4.78%
anomaly-hbos-stamps-auc REPRODUCED 0.918 0.9047 1.449%
anomaly-hbos-waveform-auc REPRODUCED 0.683 0.6879 0.717%
anomaly-hbos-wbc-auc REPRODUCED 0.982 0.9883 0.642%
anomaly-hbos-wdbc-auc REPRODUCED 0.967 0.9616 0.558%
anomaly-hbos-wpbc-auc REPRODUCED 0.536 0.5447 1.623%
anomaly-knn-breastw-auc REPRODUCED 0.976 0.9765 0.051%
anomaly-knn-cardio-auc REPRODUCED 0.724 0.7127 1.561%
anomaly-knn-ionosphere-auc REPRODUCED 0.919 0.9259 0.751%
anomaly-knn-lympho-auc REPRODUCED 0.975 0.9988 2.441%
anomaly-knn-mammography-auc REPRODUCED 0.841 0.8387 0.273%
anomaly-knn-satellite-auc REPRODUCED 0.684 0.6775 0.95%
anomaly-knn-satimage2-auc REPRODUCED 0.954 0.9364 1.845%
anomaly-knn-speech-auc REPRODUCED 0.458 0.4862 6.157%
anomaly-knn-stamps-auc REPRODUCED 0.874 0.8241 5.709%
anomaly-knn-waveform-auc REPRODUCED 0.74 0.7684 3.838%
anomaly-knn-wbc-auc REPRODUCED 0.972 0.9941 2.274%
anomaly-knn-wdbc-auc REPRODUCED 0.927 0.9992 7.789%
anomaly-knn-wpbc-auc REPRODUCED 0.526 0.5208 0.989%
anomaly-lof-cardio-auc REPRODUCED 0.574 0.5471 4.686%
anomaly-lof-internetads-auc REPRODUCED 0.606 0.6485 7.013%
anomaly-lof-ionosphere-auc REPRODUCED 0.86 0.8609 0.105%
anomaly-lof-lympho-auc REPRODUCED 0.977 0.9777 0.072%
anomaly-lof-mammography-auc REPRODUCED 0.729 0.7204 1.18%
anomaly-lof-satellite-auc REPRODUCED 0.557 0.5428 2.549%
anomaly-lof-speech-auc REPRODUCED 0.471 0.5075 7.749%
anomaly-lof-stamps-auc REPRODUCED 0.739 0.6888 6.793%
anomaly-lof-waveform-auc REPRODUCED 0.735 0.7336 0.19%
anomaly-lof-wpbc-auc REPRODUCED 0.511 0.5184 1.448%
anomaly-ocsvm-breastw-auc REPRODUCED 0.961 0.9787 1.842%
anomaly-ocsvm-cardio-auc REPRODUCED 0.935 0.9352 0.021%
anomaly-ocsvm-lympho-auc REPRODUCED 0.976 0.9906 1.496%
anomaly-ocsvm-mammography-auc REPRODUCED 0.872 0.8721 0.011%
anomaly-ocsvm-optdigits-auc REPRODUCED 0.5 0.5037 0.74%
anomaly-ocsvm-speech-auc REPRODUCED 0.447 0.4666 4.385%
anomaly-ocsvm-stamps-auc REPRODUCED 0.88 0.8517 3.216%
anomaly-ocsvm-wbc-auc REPRODUCED 0.975 0.9817 0.687%
anomaly-ocsvm-wpbc-auc REPRODUCED 0.498 0.4743 4.759%
anomaly-pca-breastw-auc REPRODUCED 0.959 0.9556 0.355%
anomaly-pca-cardio-auc REPRODUCED 0.95 0.95 0.0%
anomaly-pca-cardiotocography-auc REPRODUCED 0.755 0.7523 0.358%
anomaly-pca-hepatitis-auc REPRODUCED 0.804 0.752 6.468%
anomaly-pca-ionosphere-auc REPRODUCED 0.787 0.7841 0.368%
anomaly-pca-lympho-auc REPRODUCED 0.985 0.9965 1.168%
anomaly-pca-mammography-auc REPRODUCED 0.882 0.8938 1.338%
anomaly-pca-pima-auc REPRODUCED 0.655 0.6484 1.008%
anomaly-pca-satellite-auc REPRODUCED 0.599 0.6012 0.367%
anomaly-pca-satimage2-auc REPRODUCED 0.982 0.9772 0.489%
anomaly-pca-spambase-auc REPRODUCED 0.555 0.5503 0.847%
anomaly-pca-speech-auc REPRODUCED 0.45 0.4692 4.267%
anomaly-pca-stamps-auc REPRODUCED 0.917 0.9069 1.101%
anomaly-pca-waveform-auc REPRODUCED 0.634 0.6395 0.868%
anomaly-pca-wbc-auc REPRODUCED 0.974 0.9934 1.992%
anomaly-pca-wdbc-auc REPRODUCED 0.929 0.986 6.136%
anomaly-pca-wine-auc REPRODUCED 0.801 0.8059 0.612%
anomaly-pca-wpbc-auc REPRODUCED 0.496 0.4825 2.722%
catboost-default-vs-tuned REPRODUCED 0.27 0.2749 1.815%
ecod-breastw-auc REPRODUCED 0.993 0.9914 0.161%
ecod-cardio-auc REPRODUCED 0.897 0.935 4.236%
ecod-lympho-auc REPRODUCED 0.993 0.9953 0.232%
ecod-mammography-auc REPRODUCED 0.894 0.9062 1.365%
ecod-satimage2-auc REPRODUCED 0.985 0.9649 2.041%
ecod-shuttle-auc REPRODUCED 0.99 0.9929 0.293%
ecod-wbc-auc REPRODUCED 0.974 0.9948 2.136%
graph-deepwalk-blogcatalog-microf1 REPRODUCED 0.4151 0.4117319988879622 0.811%
graph-pygcn-cora-acc REPRODUCED 0.815 0.836 2.577%
graph-surprise-coclustering-ml100k-rmse REPRODUCED 0.963 0.97 0.727%
graph-surprise-knnbaseline-ml100k-rmse REPRODUCED 0.931 0.93 0.107%
graph-surprise-nmf-ml100k-rmse REPRODUCED 0.963 0.9626 0.042%
graph-surprise-slopeone-ml100k-rmse REPRODUCED 0.946 0.9456 0.042%
graph-surprise-svd-ml100k-rmse REPRODUCED 0.934 0.9364 0.257%
graph-vgae-cora-auc REPRODUCED 0.914 0.9094729034418444 0.495%
hdbscan-blobs-ari REPRODUCED 0.9 0.9692 7.689%
hmmlearn-gaussian-hmm REPRODUCED 0.0 0.0074 0.74%
imbalanced-learn-smote-f1 REPRODUCED 0.0 0.1724 17.24%
lightgbm-speedup-claim REPRODUCED 20.0 11.17 44.15%
ml-copod-breastw-auc REPRODUCED 0.9936 0.9944 0.081%
ml-copod-cardio-auc REPRODUCED 0.8974 0.9219 2.73%
ml-denmune-aggregation-ari REPRODUCED 0.99 0.9927 0.273%
ml-devnet-annthyroid-auc REPRODUCED 0.783 0.781 0.255%
ml-gmm-iris-ari REPRODUCED 0.9 0.9039 0.433%
ml-isoforest-digits-auc REPRODUCED 0.95 0.9865 3.842%
ml-lof-synthetic-auc REPRODUCED 0.99 0.999 0.909%
ml-pidforest-mammography-auc REPRODUCED 0.84 0.8469 0.821%
ml-pidforest-satimage2-auc REPRODUCED 0.982 0.9786 0.346%
ml-pidforest-thyroid-auc REPRODUCED 0.876 0.8825 0.742%
ml-pyod-knn-synthetic-auc REPRODUCED 1.0 1.0 0.0%
ml-quickshiftpp-blobs-ari REPRODUCED 1.0 1.0 0.0%
ml-spectral-moons-ari REPRODUCED 1.0 1.0 0.0%
ml-umap-digits-trustworthiness REPRODUCED 0.97 0.9889 1.948%
nlp-crfsuite-conll2002-f1 REPRODUCED 0.77 0.7965 3.442%
nlp-langid-identification-acc REPRODUCED 0.94 1.0 6.383%
nlp-nltk-naivebayes-movie-acc REPRODUCED 0.8 0.81 1.25%
nlp-rankbm25-retrieval-mrr REPRODUCED 1.0 1.0 0.0%
prophet-cv-mape REPRODUCED 0.1 0.0743 25.7%
river-phishing-acc REPRODUCED 0.8879 0.8928 0.552%
sb3-ppo-cartpole REPRODUCED 500.0 500.0 0.0%
sentence-transformers-sts-spearman REPRODUCED 0.85 0.8203 3.494%
sklearn-20newsgroups-tfidf REPRODUCED 0.88 0.882 0.227%
sklearn-digits-svm-acc REPRODUCED 0.97 0.9689 0.113%
ts-darts-airpassengers REPRODUCED 5.11 5.11 0.0%
ts-dlinear-etth1 REPRODUCED 0.375 0.3841443955898285 2.439%
ts-mabwiser-sim REPRODUCED 0.9 0.986 9.556%
ts-minirocket-ucr REPRODUCED 0.969 0.9602 0.908%
ts-nlinear-etth1 REPRODUCED 0.374 0.3737042248249054 0.079%
ts-pymc-coinflip REPRODUCED 0.6667 0.6654 0.195%
ts-pyro-eightschools REPRODUCED 4.4 6.597 49.932%
ts-qlearning-taxi REPRODUCED 7.9 7.9 0.0%
ts-rocket-ucr REPRODUCED 0.969 0.9611 0.815%
ts-statsforecast-m4 REPRODUCED 0.94 0.954 1.489%
xgboost-higgs-auc REPRODUCED 0.84 0.8243 1.869%
anomaly-hbos-ionosphere-auc DIVERGED 0.552 0.6421 16.322%
anomaly-lof-breastw-auc DIVERGED 0.47 0.3889 17.255%
anomaly-lof-satimage2-auc DIVERGED 0.458 0.54 17.904%
anomaly-lof-wbc-auc DIVERGED 0.932 0.8315 10.783%
anomaly-ocsvm-ionosphere-auc DIVERGED 0.838 0.7395 11.754%
anomaly-ocsvm-satimage2-auc DIVERGED 0.998 0.4209 57.826%
gensim-word2vec-analogy DIVERGED 0.6 0.2883 51.95%
graph-edmot-cora-modularity DIVERGED 0.4088 0.8151193777541244 99.393%
ml-denmune-jain-ari DIVERGED 1.0 0.2355 76.45%
ml-finch-mnist10k-nmi DIVERGED 0.8905 0.9755 9.545%
ml-kmeans-digits-nmi DIVERGED 0.74 0.6264 15.351%
nanogpt-shakespeare-gpu DIVERGED 1.47 1.8857 28.279%
nlp-glove-analogy-acc DIVERGED 75.0 23.23 69.027%
nlp-textcnn-mr-acc DIVERGED 76.1 45.3125 40.457%
node2vec-linkpred-auc DIVERGED 0.97 0.7599 21.66%
ts-statsmodels-sarimax-airline DIVERGED 1022.299 922.205 9.791%
ts-thompson-bernoulli-regret DIVERGED 21.0 10.96 47.81%
tslearn-dtw-knn-ucr DIVERGED 0.9 1.0 11.111%
umap-mnist-runtime DIVERGED 42.0 129.54 208.429%
graph-ncf-neumf-ml1m-hr10 RUN_FAILED 0.73 —
ml-suod-cardio-iforest-auc RUN_FAILED 0.9216 —
nlp-doc2vec-imdb-acc RUN_FAILED 0.87 —
nlp-fasttext-dbpedia-p1 RUN_FAILED 0.98 —
nlp-fasttext1607-agnews-acc RUN_FAILED 92.5 —
nlp-nbsvm-imdb-acc RUN_FAILED 91.55 —
nlp-sif-sts-correlation RUN_FAILED 0.717 —
nlp-vader-tweets-f1 RUN_FAILED 0.96 —
ts-pmdarima-wineind RUN_FAILED 2908.093 —
beir-bm25-anserini-ndcg BUILD_FAILED 0.65 —
graph-openne-node2vec-wiki-microf1 BUILD_FAILED 0.651 —
huggingface-bert-glue-gpu BUILD_FAILED 0.93 —
nlp-brightmart-textcnn-acc BUILD_FAILED 0.65 —
nlp-flair-sentiment-acc BUILD_FAILED 1.0 —
nlp-textcnn-mindspore-sst2 BUILD_FAILED 0.7971 —
nlp-textcnn-sst2-acc BUILD_FAILED 85.99 —
pomegranate-hmm-speedup BUILD_FAILED 13 —
pyserini-bm25-beir-ndcg BUILD_FAILED 0.679 —
ts-deeppilco-cartpole BUILD_FAILED 0.1 —
vit-pytorch-cifar-gpu BUILD_FAILED 0.88 —
graph-pygat-cora-acc TIMEOUT 0.84 —
nlp-han-agnews-acc TIMEOUT 92.7 —
nlp-scapt-absa-restaurant-acc TIMEOUT 90.0 —
nlp-simcse-sts-spearman TIMEOUT 76.25 —
ts-nbeats-m4 TIMEOUT 13.114 —
ts-nhits-ettm2 TIMEOUT 0.255 —
ts-sb3-dqn-mountaincar TIMEOUT -100.849 —

Read this number honestly