tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agorobots.txt is a suggestionusenet.lolimagemessage-square10fedilinkarrow-up11arrow-down10
arrow-up11arrow-down1imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agomessage-square10fedilink
minus-squareBlackPenguins@lemmy.worldlinkfedilinkarrow-up0·edit-25 months agoWhat else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
minus-squarelud@lemm.eelinkfedilinkarrow-up0·5 months agoIndexing is what “scrapers” mostly do. That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.
What else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
Indexing is what “scrapers” mostly do.
That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.