WEB Page Scraping EDU? v.we need one

Discussion in 'OT Technology' started by Ebtek, Mar 10, 2009.

  1. Ebtek

    Ebtek Afraid of 138

    Joined:
    Mar 19, 2000
    Messages:
    152,928
    Likes Received:
    1,091
    lets talk about scraping pages for info in this bitch.

    and where/how it can be useful.

    the little bit that i do know about it is that you can have a script check a page (even log into the page like your Aff account?) and take a snapshot of what it looks like. then log in Xminutes later ,and check it again. if its different, it will do an action like email you or whatever.

    is this right?

    can you specify a small region of the page to check it it different (i.e. the top right corner of my Maxbounty member area, as oppose to the whole page, that changes one in a while)?

    and what other applications is this good for?
     
  2. CrazyInteg

    CrazyInteg Honda-Acura.net OG

    Joined:
    Dec 30, 2000
    Messages:
    12,206
    Likes Received:
    3
    Location:
    Omaha
    I've always used perl and a regex.

    I have scripts that loop through my inventory, check competitor's prices, calculate profit, and write a report that I can access through the web. It runs like 4 times a day. That way I can keep a tab on my competitors are doing. Eventually I hope to store the data in a database and use GD::Graph to create a graph of the historical prices of my competitors.


    I started by using a lynx system command call to grab data, but I now use the LWP module.
     
  3. 2ofdem

    2ofdem OT Supporter

    Joined:
    Jun 17, 2003
    Messages:
    114,854
    Likes Received:
    8
    Location:
    aus.vic.mel
    I use Curl and Php its a bit complicated if I get some time ill write something up.
     
  4. Ebtek

    Ebtek Afraid of 138

    Joined:
    Mar 19, 2000
    Messages:
    152,928
    Likes Received:
    1,091
    fucking cutting edge for an actual store to be doing something like that on it's own :bowdown:

    i figured tecchy nerds would use this technology to set up price compairson sites, so they can make a profit on ads/referrals, but for a store to do it to keep track of the competition... :bowdown:

    what industry?
     
  5. CrazyInteg

    CrazyInteg Honda-Acura.net OG

    Joined:
    Dec 30, 2000
    Messages:
    12,206
    Likes Received:
    3
    Location:
    Omaha
    I'm just a guy selling imported product online at various websites (Ebay, etc). My full time job I'm a computer programmer, and work on web applications. So it fits together well.
     
  6. Ricky

    Ricky █▄ █▄█ █▄ ▀█▄

    Joined:
    Jun 17, 2005
    Messages:
    38,767
    Likes Received:
    6
    Last year when i was working for a marketing company, there were several tools which we used that scraped information from sites such as Monster. I'll try to take a look and see if i can get the names. They were built pretty solid. You enter in the DIV id's you want and it'll grab it + go to the next page.
     
  7. Ricky

    Ricky █▄ █▄█ █▄ ▀█▄

    Joined:
    Jun 17, 2005
    Messages:
    38,767
    Likes Received:
    6
    oh no, not just divs
     
  8. Deviance

    Deviance Can't you smell that smell? OT Supporter

    Joined:
    Jul 10, 2007
    Messages:
    4,990
    Likes Received:
    12
    Location:
    Rocky Mountain High

    I have one that logs in and grabs all of the images in the specified thread, specific to OT. I could send you the code for the login part if you want.
     
  9. Deviance

    Deviance Can't you smell that smell? OT Supporter

    Joined:
    Jul 10, 2007
    Messages:
    4,990
    Likes Received:
    12
    Location:
    Rocky Mountain High
    Ir's in perl. That an issue?
     
  10. Mikenotmike

    Mikenotmike

    Joined:
    Jun 1, 2001
    Messages:
    6,244
    Likes Received:
    0
    Location:
    USA
    Check out autoit (http://www.autoitscript.com/)

    Took a few days to learn, but you can interact with web pages in every way imaginable, scrape the info to databases or files, take screen shots, take information from one site and post it to another, it's really amazing.
     
  11. Deviance

    Deviance Can't you smell that smell? OT Supporter

    Joined:
    Jul 10, 2007
    Messages:
    4,990
    Likes Received:
    12
    Location:
    Rocky Mountain High
    Let me know how it works for ya.
     

Share This Page