this post was submitted on 29 May 2024
39 points (100.0% liked)

technology

23308 readers
364 users here now

On the road to fully automated luxury gay space communism.

Spreading Linux propaganda since 2020

Rules:

founded 4 years ago
MODERATORS
Jadzia_Dax@hexbear.net
context@hexbear.net
EmmaGoldman@hexbear.net
SexUnderSocialism@hexbear.net
gaycomputeruser@hexbear.net
ZoomeristLeninist@hexbear.net
39
Should Hexbear have a more robust robots.txt? (hexbear.net)
submitted 5 months ago by neo@hexbear.net to c/technology@hexbear.net
18 comments fedilink hide all child comments
 

Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.

you are viewing a single comment's thread
view the rest of the comments
[–] neo@hexbear.net 10 points 5 months ago (4 children)

Of course it's voluntary, but if entities like OpenAI say they will respect it then presumably they really will.

[–] vk6flab@lemmy.radio 18 points 5 months ago (1 children)

Couple of things:

  1. Do you believe anything coming out of OpenAI when it's abundantly clear that they'll say anything to protect their bottom line.
  2. OpenAI are not the only people harvesting data and selling it to interested parties.
  3. There is no legal requirement to adhere to the standard and I'd be shocked if any court in the USA could understand the issue, let alone enforce a voluntary standard.
  4. The amount of automated data collection online is staggering. On my own services it accounts for 50% of the hits. Good luck with policing that.
[–] neo@hexbear.net 7 points 5 months ago (2 children)

I agree with your points 2-4 but I have observed on my own website that the crawlers who don't respect won't, and the crawlers who do respect will.

[–] the_itsb@hexbear.net 6 points 5 months ago (1 children)

How did you find this information? I know how to check traffic for my website, but idk how to get from "list of IPs" to "these ones are crawlers"

apologies if this is a silly question

[–] neo@hexbear.net 2 points 5 months ago* (last edited 5 months ago)

I used to sit and monitor my server access logs. You can tell by the access patterns. Many of the well-behaved bots announce themselves in their user agents, so you can see when they're on. I could see them crawl the main body of my website, but not go to a subdomain, which is clearly linked from the homepage but is disallowed from my robots.txt.

On the other hand, spammy bots that are trying to attack you will often instead have access patterns that try to probe your website for common configurations for common CMSes like WordPress. They don't tend to crawl.

Google also provides a tool to test robots.txt, for example.

[–] vk6flab@lemmy.radio 1 points 5 months ago

Perhaps this will help your understanding of my first point.

https://gizmodo.com/former-openai-board-member-sam-altman-chatgpt-1851506252

[–] nat_turner_overdrive@hexbear.net 11 points 5 months ago

presumably they really will

but why presume that the guys trying to scam people by claiming their algorithms are aRtIfIcIaL iNtElLiGeNcE aren't lying about that?

[–] space_comrade@hexbear.net 4 points 5 months ago

but if entities like OpenAI say they will respect it then presumably they really will.

Eh, will they really? It'd be pretty hard to prove they didn't respect it.