I’m getting really tired of “verify you’re not a bot” checks everywhere.

I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?

  • taiyang@lemmy.world
    link
    fedilink
    arrow-up
    162
    ·
    9 days ago

    No, but only because I’ve helped run a website recently and you underestimate just how much wasted traffic is from scrapper bots. On our limited bandwidth, it was practically a DDOS attack all the time.

    • mesa@piefed.social
      link
      fedilink
      English
      arrow-up
      39
      ·
      9 days ago

      My solution reduced 95ish percent of traffic. Bots were hitting my tiny site like mad.

      • Scrubbles@poptalk.scrubbles.tech
        link
        fedilink
        English
        arrow-up
        53
        ·
        edit-2
        9 days ago

        I went from thousands of requests per minute to 10, tops. Bot prevention is a must have.

        If bot checks were removed then everyone would be complaining that it’s unusably slow.

        • ElectricVocalist@jlai.lu
          link
          fedilink
          arrow-up
          1
          arrow-down
          17
          ·
          9 days ago

          It’s easy to block most bots. It’s impossible to block bots that target you specifically. I’m not blocking any bots on my websites, I like it better to optimize performance so that I can serve them all and have a lot of headroom

          • GenosseFlosse@feddit.org
            link
            fedilink
            arrow-up
            17
            ·
            edit-2
            9 days ago

            No, it’s not. Had request from a bot network, always changing ip’s, user agent and request url. There is no pattern there I can block.

            • mesa@piefed.social
              link
              fedilink
              English
              arrow-up
              16
              ·
              edit-2
              9 days ago

              I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.

              • GenosseFlosse@feddit.org
                link
                fedilink
                arrow-up
                12
                ·
                9 days ago

                No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.

                • America@sh.itjust.works
                  link
                  fedilink
                  arrow-up
                  9
                  arrow-down
                  1
                  ·
                  9 days ago

                  That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.

        • mesa@piefed.social
          link
          fedilink
          English
          arrow-up
          4
          ·
          9 days ago

          Python script with fail2ban. Registers invalid requests then blocks. Works well.

      • PapaStevesy@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        8 days ago

        Is there a way to trick the bots into thinking your site was made by an LLM so they won’t want to train on it?

        • urushitan 漆たん@kakera.kintsugi.moe
          link
          fedilink
          arrow-up
          1
          ·
          edit-2
          8 days ago

          The easiest way would likely be to add an ai written html comment attributing a real ai model and version to the content creation, with a comment that all of the info on the page was ai generated, anecdotal and hadn’t been researched. An ingestion model is then likely to ignore it or rate it low. It should be noted too that all the big ai companies now honor robots.txt and LLMs.txt, so that would more likely be defending against “neolabs” and international labs who ignore them.

  • WolfLink@sh.itjust.works
    link
    fedilink
    arrow-up
    61
    ·
    9 days ago

    You may not understand just how prevalent bot traffic is. I self-host a small personal website that real people only actually visit once in a blue moon, but I get constant attempts to access the Wordpress admin console (my website isn’t made with Wordpress lol)

  • WormFood@lemmy.world
    link
    fedilink
    arrow-up
    55
    ·
    8 days ago

    In the last 24 hours my small personal website got 8 legitimate visitors and 74,000 requests from bots, most of which were either malicious or aggressive ai crawlers

  • Hal@piefed.nz
    link
    fedilink
    English
    arrow-up
    49
    arrow-down
    1
    ·
    9 days ago

    As others have said, they are absolutely necessary in this day and age.

    But also the bots are not deliberately targetting lemmy, it’s just that lemmy has unlimited pages to scrape and so the traffic piles up. If your issue is specifically with Lemmy then using an app should help. Many instances also give logged in users a free pass.

    • ᴋɪᴋɪ@lemmy.worldOP
      link
      fedilink
      arrow-up
      5
      ·
      edit-2
      9 days ago

      I could use an app, but since I don’t know English, I need to use a translator to understand what people are writing.

      • Hal@piefed.nz
        link
        fedilink
        English
        arrow-up
        9
        ·
        edit-2
        9 days ago

        Ah well I’d suggest for Lemmy make sure you’re logged in. Many instances will bypass checks if the right cookie is set. Not always the case, as each instance would need to implement the bypass themselves. I’m not sure if Lemmy.world has done this.

        • ChunkMcHorkle@lemmy.world
          link
          fedilink
          English
          arrow-up
          10
          ·
          9 days ago

          On browser at least, lemmy.world – OP’s instance and my own – only requires a bot check if you’re not logged in, even if you’re behind a VPN (I am). I never see a bot check unless I am going in via a different browser or device where I am not logged in.

          • Hal@piefed.nz
            link
            fedilink
            English
            arrow-up
            9
            ·
            9 days ago

            Cool, thanks for confirming.

            Bot checks will only be for the website in a browser. Apps connect to the API, and you can’t really put a visible check in a silent background request.

            The fediverse also is built on computers talking to each other, things like cloudflares bot blocking will sometimes kill federation because ultimately it’s bots talking to bots. But it would make no sense to connect to the API for scraping, if you are specifically targeting lemmy by calling its API, then you might as well just make your own instance, and then all the other instances will freely send you everything.

            • ChunkMcHorkle@lemmy.world
              link
              fedilink
              English
              arrow-up
              4
              ·
              9 days ago

              Makes sense. I knew bot traffic is an immense problem but this thread is up to date and has been informative. Thanks for explaining.

  • chunes@lemmy.world
    link
    fedilink
    arrow-up
    41
    ·
    9 days ago

    I don’t like it, but I will say that it beats that period where I was forced to constantly do captchas

  • Hemingways_Shotgun@lemmy.ca
    link
    fedilink
    English
    arrow-up
    34
    ·
    8 days ago

    I’ve just considered it a casualty of the privacy wars.

    I don’t know if they’re deployed as a “weapon of annoyance” or anything, but I do notice that when I’m using a) a degoogled browser, b) an ad-blocking dns, c) mask my location, and/or d) block my IP. Suddenly (almost) every website makes me wait by throwing that at me.

    I think it’s lowkey trying to annoy me to the point where I give up on having those privacy measures, but they underestimate my hatred of corporate fuckery.

    • fdnomad@programming.dev
      link
      fedilink
      English
      arrow-up
      8
      ·
      8 days ago

      I’d guess your client just looks more like a crawler that way and gets a JS challenge thrown its way

  • LeftReddit2@lemmy.world
    link
    fedilink
    arrow-up
    31
    ·
    8 days ago

    These days? Fuck no. I think it’s like 57% of all web traffic these days is just bots. This is absolutely needed.

    • twistabledeer@lemmy.world
      link
      fedilink
      arrow-up
      15
      ·
      8 days ago

      Cloudflare is getting rich from all of these bots. We had turn to Cloudflare at work because bots were scraping our website and essentially DDoSing. Cloudflare worked perfectly and rate limiting bot scraping.

      • JigglySackles@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        7 days ago

        I wouldn’t be surprised if they make some of the bots they stop. It’d be like the people who bred snakes to get money for killing them in India.

    • rumba@lemmy.zip
      link
      fedilink
      English
      arrow-up
      8
      ·
      edit-2
      8 days ago

      I run a reticulum NomadNet site. I setup logging, since there are no ‘sessions’ and I don’t want to force everyone to identify, I just calcuate bot traffic based on time, order and frequency of loads, bots’s scraping are likely 80% of my traffic. And this is for a service that’s designed to support 200bps radio links.

    • froh42@lemmy.world
      link
      fedilink
      arrow-up
      11
      arrow-down
      2
      ·
      8 days ago

      I’m even more annoyed by anubis. Letting me wait in the browser for an enternity for some check that a bot with a native implementation can solve in milliseconds.

      Proof of work my ass.

      • dev_null@lemmy.ml
        link
        fedilink
        arrow-up
        7
        ·
        8 days ago

        It sure can, instead of doing 50 requests. A slowdown is all that’s needed to make random scraping uneconomical.

        • froh42@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          edit-2
          8 days ago

          https://github.com/mikenrafter/anubis-fast.mirror

          This is a sample of a much faster anubis solver (end user facing).

          There’s versions. of this for people running bot networks - it’s just a few cheap cloud cpu cycles to invest. Proof of work is an ineffective defense against spam in the age of cheap and scalable cloud compute - and even moreso if the botnet runs on residential proxies.

          My point is, the slowdown is NOT making random scraping uneconomical anymore, it does hurt legitimate users a lot more.

          • Otter@lemmy.ca
            link
            fedilink
            arrow-up
            1
            ·
            2 days ago

            From what we are seeing, it’s currently helping on our instances. I see the argument that it might stop working someday, but for now it’s significantly cut down on bad traffic. The bots tend to buzz off and bother someone else

      • Axolotl.cpp@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        4
        ·
        edit-2
        8 days ago

        Tbh I never had to wait much with anubis, neither with my phone or computer; websites that make me wait a lot (like the arch wiki IIRC) then give ya access without having to do the check again;
        Also, Anubis unlike Cloudflare does not block me just cuz i have NoScript installed

        • froh42@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          8 days ago

          How modern is your phone? Anubis is killing my old Samsung A72 from 2020 - long waits, draining battery. On the laptop it’s less of an annoyance, I’ve got the laptop connected to mains anyway and it is fast enough.

  • GenosseFlosse@feddit.org
    link
    fedilink
    arrow-up
    25
    ·
    9 days ago

    Have a small website, and the bot traffic is overwhelming. Was slowing down everything, exhaust the resources I was allowed from my hoster and try to access non existent pages. Bots have ruined the Internet.

  • Arcanepotato@crazypeople.online
    link
    fedilink
    English
    arrow-up
    25
    ·
    edit-2
    9 days ago

    A lot of the comments seem to be focused on AI bots or scraping.

    These are intended to protect against super high volumes of requests that drain resources. Like others have mentioned, it may be related to a VPN. The system is detecting a lot of requests from a single IP. If you are on a VPN or using another tool to spoof your IP you might be coming from the same IP as other people who are accessing it at the same time or have recently tried to access it. Website and other services usualy have bandwidth caps so they can only serve so many responses at once, and if are all being taken up by bots, the website or other service won’t work for legit users. This is how a DDoS attack works. Eta: so these checks are less about stopping bots from accessing the site than stopping bots from making the site inaccessible to legit users.

    (This is a very simplified explanation because that’s my level of understanding, I hope if there are errors someone will jump in and correct.)

    Do I find it annoying? Yes. But I understand the site might not work without them.

    • ᴋɪᴋɪ@lemmy.worldOP
      link
      fedilink
      arrow-up
      11
      arrow-down
      9
      ·
      9 days ago

      Fuck, so now anyone who’s different is a bot or AI? Seriously, if I were a robot, why would I write a post complaining about websites? If I were AI, I’d have better things to do.

        • ᴋɪᴋɪ@lemmy.worldOP
          link
          fedilink
          arrow-up
          2
          arrow-down
          16
          ·
          9 days ago

          Dude, you’re watching too many science fiction movies. Even though artificial intelligence might be quite advanced these days, if I were a robot I wouldn’t waste my time talking to humans. I’d think of something worse: ending the lives of these poor, brainless mortals. You’re crazy, man :-P

  • CanadaPlus@lemmy.sdf.org
    link
    fedilink
    arrow-up
    21
    ·
    edit-2
    8 days ago

    It’s proportional to the amount of rude AI crawling, from what I’ve heard.

    The industry has put every reasonably accessible piece of human writing in their datasets, by now, and are scrounging for the remaining little bits. Old, obscure, obsolete paper books nobody ever reads are apparently a hot item. So, yeah, your latest blog post is probably worth some fraction of a penny, and that’s all it takes.

          • 🇨🇦Samuel Proulx🇨🇦@rblind.com
            link
            fedilink
            English
            arrow-up
            3
            ·
            8 days ago

            Oh, as long as it never displays a page that requires user interaction, it works. If you go to rblind.com directly, you’ll find I’m doing something similar. I didn’t use anubis because of some controversy about the logo, and the logo has no alt text for me to judge if I care about whatever upsets people. So I just decided not to use it.

            • Axolotl.cpp@lemmy.dbzer0.com
              link
              fedilink
              arrow-up
              4
              ·
              8 days ago

              I’il check out! Anyway, if you are curious: the anubis logo is just a cartoon of a girl with cat ears and a magnifying glass, nothing too crazy beside it looking “unprofessional”

              • grysbok@lemmy.sdf.org
                link
                fedilink
                English
                arrow-up
                4
                ·
                8 days ago

                Agreed re: the logo. And she has a maple leaf symbol on her white hat because she’s Canadian.

                The creator also has a paid version where you can change the logo. The creator is a chill person.

                • grysbok@lemmy.sdf.org
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  8 days ago

                  She’s in a grey hoodie and red/pink skirt (image ends above her knees, so you don’t see much of the skirt). She would look at home in a children’s animated series. “Unprofessional”, but not scandalous.

              • Carl@anarchist.nexus
                link
                fedilink
                English
                arrow-up
                3
                ·
                8 days ago

                Minor nit-picky correction: She’s a jackal, not a cat. So she’s technically more like a foxgirl, instead of a catgirl.

                Because Anubis is the jackal-headed Egyptian god that weighs your guilt at the end of your life. And if your guilt weighs less than a feather, you’re allowed to enter the Egyptian equivalent of heaven.

                The reference being that the service is “weighing” the users, and only allowing the guilt-free ones (humans) to access heaven (the site).