RE: https://tldr.nettime.org/@tante/116605858023186072
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
@inthehands I found a google note on meta tag so saying that if you use robots.txt to exclude googlebot, it won’t be able to crawl your pages to honor noindex, so if another site google does index points to a page on your site, the link will still be indexed.
https://developers.google.com/search/docs/crawling-indexing/block-indexing
-
Defeatism is a form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands annoy people into doing better.
-
@inthehands As I said just a while ago: Every big tech press event these last few years have felt like "Announcing our exciting plans for oligarchs to strip-mine the entire world and immiserate all of humanity! Get on board, and also death to the unbelievers!"
@inthehands@hachyderm.io @datarama@hachyderm.io
Here's how I've seen the response to Google's latest bullshit:
-
@glassresistor @inthehands are you doing this in a particular way? Basically looking for different approaches.
@rooneymcnibnug @inthehands filtering by user agent, ip address, cloudfront no bots acl config, by load
-
@rooneymcnibnug @inthehands filtering by user agent, ip address, cloudfront no bots acl config, by load
@glassresistor @inthehands oh okay I was misinterpreting the statement, my bad
-
@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery. -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands hi! Did anyone answer to your question here? I'm very interested in blocking the bots, but everyone seems so eager to give their fucking opinions instead of answering this, that I couldn't find a proper way of blocking said bots. Thanks!
-
@gherhartd @inthehands Use Brave browser and Brave search, or equivalent, might be one way
@commons_protocol @inthehands the problem isn't our own usage, the problem is the hegemonic place of Chrome. Forbidding Chrome browser from accessing one's site is almost completely cutting it from the web, from nearly everybody.
-
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
@joe @ShadSterling @inthehands there is one in Java?
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands all the bots that are used to train AI, don't care about robots.txt, or meta stuff. Otherwise, it will be much more harder to train stuff.
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
I’m mostly worried about their “agentic” part, because that sounds like new infrastructure with possibly different user agents etc., so harder to ban, and I’m
sure it will DEFINITELY have no “social contract” whatsoever. -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands
Some websites are being deindexed but they're still being crawled by Google
https://social.wikidex.net/@ciencia/116611028083997779 -
@korrupt @inthehands is “Meta noindex” Like “swiper, no swiping”?
@amyworrall @inthehands I don't get the "Swiper, no swiping"..?
The effect is "Hey, here you see a page which must not be indexed in google search" -> google sees it, reads it and won't index it, regardless of other signals (links, sitemap etc.). robots.txt forbids access to the page. -> Google knows nothing about page content, but may index the URL, if its found elsewhere (external link, sitemap, whatever). My experience: with no further information -> You get a very ugly SERP snippet
-
@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery. -
M monkee@chaos.social shared this topic
