I’m getting really tired of “verify you’re not a bot” checks everywhere.

I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?

  • mesa@piefed.social
    link
    fedilink
    English
    arrow-up
    39
    ·
    9 days ago

    My solution reduced 95ish percent of traffic. Bots were hitting my tiny site like mad.

    • Scrubbles@poptalk.scrubbles.tech
      link
      fedilink
      English
      arrow-up
      53
      ·
      edit-2
      9 days ago

      I went from thousands of requests per minute to 10, tops. Bot prevention is a must have.

      If bot checks were removed then everyone would be complaining that it’s unusably slow.

      • ElectricVocalist@jlai.lu
        link
        fedilink
        arrow-up
        1
        arrow-down
        17
        ·
        9 days ago

        It’s easy to block most bots. It’s impossible to block bots that target you specifically. I’m not blocking any bots on my websites, I like it better to optimize performance so that I can serve them all and have a lot of headroom

        • GenosseFlosse@feddit.org
          link
          fedilink
          arrow-up
          17
          ·
          edit-2
          9 days ago

          No, it’s not. Had request from a bot network, always changing ip’s, user agent and request url. There is no pattern there I can block.

          • mesa@piefed.social
            link
            fedilink
            English
            arrow-up
            16
            ·
            edit-2
            9 days ago

            I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.

            • GenosseFlosse@feddit.org
              link
              fedilink
              arrow-up
              12
              ·
              9 days ago

              No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.

              • America@sh.itjust.works
                link
                fedilink
                arrow-up
                9
                arrow-down
                1
                ·
                9 days ago

                That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.

                • GenosseFlosse@feddit.org
                  link
                  fedilink
                  arrow-up
                  7
                  ·
                  9 days ago

                  I already looked into something similar, IP banning does not work. One day i got ~120k unique IPs, but the most active one only made ~100 requests. I would gain nothing if i run any script to lookup the IP with each requests (and using up more resources in the process), if it fails 99% of the time because the next hit is a different IP.

                  • mesa@piefed.social
                    link
                    fedilink
                    English
                    arrow-up
                    4
                    ·
                    edit-2
                    9 days ago

                    Ok. Sounds like your in a unique situation. Good luck. You seem fucked.

                    All the above worked for me.

      • mesa@piefed.social
        link
        fedilink
        English
        arrow-up
        4
        ·
        9 days ago

        Python script with fail2ban. Registers invalid requests then blocks. Works well.