RE: https://tldr.nettime.org/@tante/116605858023186072
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
-
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
-
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
@inthehands TIL thanks
-
@inthehands TIL thanks
-
@inthehands As I said just a while ago: Every big tech press event these last few years have felt like "Announcing our exciting plans for oligarchs to strip-mine the entire world and immiserate all of humanity! Get on board, and also death to the unbelievers!"
-
@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff@AccordionBruce @markwyner @inthehands That happened about five years ago.
-
Defeatism is a form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands My site is tiny, but I had to make sure and add that into my own. I don't care if Google never happens to come across it, the idea of them using my site to train their LLM is sickening.
I also have to say, your site is absolutely gorgeous. -
@inthehands
How long until Maps will only give directions in real-time, no destination foreknowledge allowed.@sollat @inthehands They do that already by recalculating and finding the most efficient route regardless of what you tell it to do. Google knows best. Pay no attention to the sign that says "turn back now. there has never been a bridge across this 1500 foot deep canyon."
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands honestly why would anyone still care about SEO when Google doesn't forward people to your website.
Might as well block the googlebot in your robots.txt tbh
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands in forcing that fight, google is going to find that the rest of the internet already has sophisticated tools for this fight. My anubis config should already be blocking google.
-
@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands completely unhinged and fact-free question from the back of the crowd here:
How long until they no longer give a fuck about and declare robots.txt an obsolete technology? -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands for the last ~ 18 months or so before I shut my websites down last year I just served never-ending gzip streams of null bytes to all of Google’s IP space (and anyone else identifying themselves as a crawler more generally, among a few other criteria). It didn’t take very long for Google to drop the index listing.
In particular OpenAI’s dumbass scraper just kept coming back every day for gigabytes and gigabytes of “robots.txt” and other URLs, getting a 10:1 bandwidth expansion out of gzip was quite satisfying
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands
We need to get together a federated search index. -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.
1/2
@inthehands Google: how do I degoogle?
-
@inthehands Google: how do I degoogle?
-
@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.
-
@inthehands I fear that when we block the bot, they'll access the site through Chrome. Getting what people browsing get. I don't know what could stop them.
@gherhartd @inthehands Use Brave browser and Brave search, or equivalent, might be one way
-
@sollat @inthehands They do that already by recalculating and finding the most efficient route regardless of what you tell it to do. Google knows best. Pay no attention to the sign that says "turn back now. there has never been a bridge across this 1500 foot deep canyon."
-
@inthehands meta noindex it is, definitely. robots disallow can actually hurt the process, since google cannot access the file with the noindex header and therefore won't deindex.
btw, they do indeed respect noindex and robots.txt ATM, since its qute easy to check if pages still get found. Then again, you never know what does not show up in search but is used for training (without giving credit, obv.) anyway. As far as i see, google still remains more standard compliant as e.g. OpenAI.
