this post was submitted on 28 Nov 2024
960 points (99.2% liked)
Technology
60007 readers
2807 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Yeah guys. Downvote me. This is literally part of my day job.
In our industry we call not using an official API a dumpster fire API. Because more than once it has completely broken eventually, and there are a few manufacturers warned will break in the upcoming future
Is it illegal to scrape? I mean most companies and governments scrape huge amounts of data all the time without getting anyones permission, hell the entire business of a search engine is basically WeScrape4U.
If you extract a key from a proprietary client to decrypt a private API, it definitely is illegal. If you do this as a corporation, it would also be easy to detect by abnormal usage patterns tied to your account and presents you as a nice big set of pockets to get sued.
Just screen scraping isn't illegal, but is fragile as fuck and will break at the worst time and you have no ability to implement version control to manage it.
Personally I wouldn't want to be the guy forced to write scrapers with how much they break on any modern site