Here is a list of bad bots that spider your site

Discussion in 'OT Technology' started by kinyo, Jan 7, 2005.

  1. kinyo

    kinyo Guest

  2. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    it definitely doesn't hurt to do that... But beware that not all bots play nice -- and some will still spider you... So it's a good extra measure, but don't rely on it to be your only means of privacy.
     
  3. Juvenall

    Juvenall What Would Juvie Do?

    Joined:
    Dec 31, 2004
    Messages:
    2,221
    Likes Received:
    0
    Location:
    #!/usr/bin/Detroit
    Very true. When I notice a rogue bot on in my site's logs, I will block the IP at the server level. In the end, that's really the best you can do against those things.
     
  4. Moonwalker

    Moonwalker fagulous

    Joined:
    Jan 5, 2005
    Messages:
    502
    Likes Received:
    0
    Location:
    Atlanta GA
    dumb n00b question here but what exactly do the "bad bots" do? Lag your site?
     
  5. Rob

    Rob OT Supporter

    Joined:
    Jul 6, 2002
    Messages:
    88,631
    Likes Received:
    41
    Location:
    Atlanta, GA
    Besides finding email address I don't see what is bad about having them crawl your site?

    You shouldn't be posting a good email address on the web anyways.
     
  6. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    linking to files and leeching is prolly the biggest problem.
     
  7. Rob

    Rob OT Supporter

    Joined:
    Jul 6, 2002
    Messages:
    88,631
    Likes Received:
    41
    Location:
    Atlanta, GA
    If you can't handle the bandwidth don't make the file accessible. :noes:
     
  8. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    I've got 100Mbps uplinks on all my boxes... But when a file gets out in the open, it's not impossible for my servers to get 30+Mbps of sustained transfer.... And that adds up on the bill real quick!
     

Share This Page