arxiv API 429

[Archie-Zhang321]

问题描述

在send emails daily过程中经常出现报错：arxiv API 429 on batch 0，并且在多次尝试后依然失败，如图

具体信息如下： Traceback (most recent call last): File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/__init__.py", line 695, in _parse_feed return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/__init__.py", line 732, in __try_parse_feed raise HTTPError(url, try_index, resp.status_code) arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 503 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/src/zotero_arxiv_daily/main.py", line 32, in main
executor.run()
~~~~~~~~~~~~^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/src/zotero_arxiv_daily/executor.py", line 102, in run
papers = retriever.retrieve_papers()
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/src/zotero_arxiv_daily/retriever/base.py", line 25, in retrieve_papers
raw_papers = self._retrieve_raw_papers()
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/src/zotero_arxiv_daily/retriever/arxiv_retriever.py", line 142, in _retrieve_raw_papers
batch = list(client.results(search))
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 648, in _results
feed = self._parse_feed(page_url, first_page=True)
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 703, in _parse_feed
return self._parse_feed(url, first_page=first_page, _try_index=_try_index + 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 703, in _parse_feed
return self._parse_feed(url, first_page=first_page, _try_index=_try_index + 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 703, in _parse_feed
return self._parse_feed(url, first_page=first_page, _try_index=_try_index + 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 705, in _parse_feed
raise err
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 695, in _parse_feed
return self.__try_parse_feed(url, first_page=first_page, try_index=_try_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/zotero-arxiv-daily/zotero-arxiv-daily/.venv/lib/python3.13/site-packages/arxiv/init.py", line 732, in __try_parse_feed
raise HTTPError(url, try_index, resp.status_code)
arxiv.HTTPError: Page request resulted in HTTP 429 (https://export.arxiv.org/api/query?search_query=&id_list=2605.15202v1%2C2605.15204v1%2C2605.15205v1%2C2605.15206v1%2C2605.15207v1%2C2605.15208v1%2C2605.15215v1%2C2605.15217v1%2C2605.15218v1%2C2605.15219v1%2C2605.15224v1%2C2605.15227v1%2C2605.15228v1%2C2605.15231v1%2C2605.15235v1%2C2605.15239v1%2C2605.15240v1%2C2605.15242v1%2C2605.15243v1%2C2605.15246v1&sortBy=relevance&sortOrder=descending&start=0&max_results=100)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.>

之前运行test, send email daily均有成功的情况，但是429现象频繁，请教作者有没有好的解决办法

[TideDra]

更新代码

[Archie-Zhang321]

感谢作者大大回复，但是我已经更新到新版代码了，还是偶尔会出现这个问题

更新代码

[Banana889]

我也遇到了这个问题。我推测原因是 executor.py 中 retrive papers 和之前对网站的访问（fetch paper）之间由于没有时间间隔，导致被 arxiv 判定为频繁，即使 retrieve_papers 内部设置了 delay_seconds ，但是只保证了两批 retrive paper 之间的间隔，没有保证第一次retrive和前面 fetch 直接的时间间隔，所以我的解决方案是在 retrive 之前等待一下，你可以试试
executor.py：

time.sleep(11)  # 新加代码
logger.info(f"Retrieving {source} papers...")
papers = retriever.retrieve_papers()

---

二编：
这个方法似乎不起作用，我又研究了一下，觉得可能是太多人同时通过 github action 发请求，导致被 arxiv 限制了，我尝试把发请求的时间提前了一点，并且代码更新到了最新版，过几天看看还有没有问题。orz

[shaashou]

同样的问题

[callme110]

me too， I still have the same problem after updating the code, API current limiting.