RE: https://tldr.nettime.org/@tante/116605858023186072
-
@gherhartd @inthehands Use Brave browser and Brave search, or equivalent, might be one way
Brave’s leadership is too personally despicable for me to consider their products
-
@inthehands My site is tiny, but I had to make sure and add that into my own. I don't care if Google never happens to come across it, the idea of them using my site to train their LLM is sickening.
I also have to say, your site is absolutely gorgeous. -
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands Google executives don't care about consent. Google executives care about their bottom line.
Copyright class action seems most promising. Along with adversarial material to poison their scrapers
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands How to keep Google from indexing your website
Official docs from Google:
<https://developers.google.com/search/docs/crawling-indexing/block-indexing>Other ways:
<https://exertpro.com/how-can-i-keep-google-from-indexing-my-website/>Other specific search engines may have their own tools for blocking their engine from indexing a web site or specific pages within it.
-
@inthehands
"Get in, loser. We're doing web rings again."@jargoggles @inthehands finally my chance to join the "ate my balls" web ring
https://iowastatedaily.com/218801/uncategorized/eating-balls-on-the-internet/
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands In that case, it's better if we send them poisoned data instead, using iocaine [0] or nepentheses [1].
[0]: https://iocaine.madhouse-project.org
[1]: https://zadzmo.org/code/nepenthes/ -
Defeatism is a form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands I needed to read this today.
-
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
@ShadowJonathan @inthehands what if we honeypotted them? Get them to ingest gigabytes upon gigabytes of LLM generated text to ruin their.efforts?
-
@inthehands you can block their bots at the network level
@glassresistor @inthehands are you doing this in a particular way? Basically looking for different approaches.
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands I found a google note on meta tag so saying that if you use robots.txt to exclude googlebot, it won’t be able to crawl your pages to honor noindex, so if another site google does index points to a page on your site, the link will still be indexed.
https://developers.google.com/search/docs/crawling-indexing/block-indexing
-
Defeatism is a form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands annoy people into doing better.
-
@inthehands As I said just a while ago: Every big tech press event these last few years have felt like "Announcing our exciting plans for oligarchs to strip-mine the entire world and immiserate all of humanity! Get on board, and also death to the unbelievers!"
@inthehands@hachyderm.io @datarama@hachyderm.io
Here's how I've seen the response to Google's latest bullshit:
-
@glassresistor @inthehands are you doing this in a particular way? Basically looking for different approaches.
@rooneymcnibnug @inthehands filtering by user agent, ip address, cloudfront no bots acl config, by load
-
@rooneymcnibnug @inthehands filtering by user agent, ip address, cloudfront no bots acl config, by load
@glassresistor @inthehands oh okay I was misinterpreting the statement, my bad
-
@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery. -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands hi! Did anyone answer to your question here? I'm very interested in blocking the bots, but everyone seems so eager to give their fucking opinions instead of answering this, that I couldn't find a proper way of blocking said bots. Thanks!
-
@gherhartd @inthehands Use Brave browser and Brave search, or equivalent, might be one way
@commons_protocol @inthehands the problem isn't our own usage, the problem is the hegemonic place of Chrome. Forbidding Chrome browser from accessing one's site is almost completely cutting it from the web, from nearly everybody.
-
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
@joe @ShadSterling @inthehands there is one in Java?
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands all the bots that are used to train AI, don't care about robots.txt, or meta stuff. Otherwise, it will be much more harder to train stuff.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
I’m mostly worried about their “agentic” part, because that sounds like new infrastructure with possibly different user agents etc., so harder to ban, and I’m
sure it will DEFINITELY have no “social contract” whatsoever.
