I've tried scraping arXiv before and they blocked my access to the website stating suspicious activity and that I should contact arXiv owners if I want to scrape it, despite the reason for arXiv existing being for Open Access to scientific articles.

Scraping for scientific articles is still on my plans, but not arXiv anymore.

The gap between the common user and the big technology players is more than just a gap of knowledge, these agreements to keep everything on the hands of big companies is problematic, especially when dealing with important philosophical concepts that should guide some websites, like Open Science.

But it is what it is, no use sending an email to arXiv telling they're wrong and they should change their minds. I'll have to look for options.

[–] e0qdk@reddthat.com 12 points 1 month ago

arXiv has bulk access methods -- you shouldn't need to scrape their website to get the data: https://info.arxiv.org/help/bulk_data.html

If you really want everything (5TB+), that's available from their S3 bucket if you're willing to cover the transfer costs: https://info.arxiv.org/help/bulk_data_s3.html