WEB if i have robots.txt do i need to have meta robot tags in my html

Discussion in 'OT Technology' started by pharmokan, Sep 29, 2009.

  1. pharmokan

    pharmokan OT Supporter

    Joined:
    Oct 18, 2002
    Messages:
    102,083
    Likes Received:
    400
    Location:
    L.A.
    i think this is a case of precedence... robots look for robots.txt first

    right
     
  2. hank85

    hank85 sudo shred /dev/sda -f -v -z --iterations=6

    Joined:
    Jul 23, 2008
    Messages:
    4,360
    Likes Received:
    0
    Correct, sir.
     
  3. pharmokan

    pharmokan OT Supporter

    Joined:
    Oct 18, 2002
    Messages:
    102,083
    Likes Received:
    400
    Location:
    L.A.
    so it doesnt matter if i have robot meta tags in my html right?

    :bowdown:
     
  4. hank85

    hank85 sudo shred /dev/sda -f -v -z --iterations=6

    Joined:
    Jul 23, 2008
    Messages:
    4,360
    Likes Received:
    0

    Not at all. Bot checks robots.txt then goes on to check metas on a per page bases. Use robots meta tags when you want to deny indexing.

    noindex,nofollow, etc.
     
  5. illectronic

    illectronic I'm Coming Home OT Supporter

    Joined:
    Jan 19, 2005
    Messages:
    16,950
    Likes Received:
    0
    Location:
    Brooklyn
    YOU MAD DUMB SON
     

Share This Page