Portia Scraper for Porn

Portia is a tool for visually scraping web sites without any programming knowledge. In theory, you just point and click on a web page to indicate what data you want to extract, and portia will learn how to scrape similar pages from the site.

Point and click? No undocumented command-line options? WTF! I spilled some of that delicious Mountain Dew to get rid of the foul taste in my mouth. Why did I even get out of bed this morning? Not that I usually leave my bed in the morning, but what made me read the installation guidelines? Life certainly could not get any worse than this.

Seven seconds later everything was fine again: I went back to basics and thought of the one thing why we geeks love technology: access to porn. Portia could save years of manual classification and indexing work! Point and click definately would be fun.

My thoughts went wild, maybe I could even integrate this with Notify My Android and automate sending my credit-card details to any new friend? Index TaoBao as a birthday gift for my mother?

Nah, let's go with the original plan, install the beast and get into some indecent indexing.

I used a random 5 USD Ubuntu VPS at DigitalOcean. This is just perfect for some nasty Linux plumbing. Log into the Ubuntu box and make sure that mother cannot hack into your server:

 sudo apt-get update
 sudo apt-get -y upgrade

I suppose you already have the nice developer tools installed, if uncertain throw this at the command prompt:

 sudo apt-get -y install git-core curl python libssl-dev build-essential

Copy and install the software:

 cd
 git clone https://github.com/scrapinghub/portia.git
 cd portia
 cd slyd
 sudo pip install -r requirements.txt

Ridiculously easy! Start the web-server with these commands:

 cd
 cd portia
 cd slyd
 twistd -n slyd

You find the website at:

http://localhost:9001/static/main.htm

Change the localhost to the address of your VPS. You might need to modify some firewall settings.

That is is, this should fire up Portia. Lock the door, enter the site URL of the site to crawl and start having fun.

Screenshot of Portia start-up screen

Herman Stevens

Just some guy on the internet. Loves technology, diving, travelling, photography and Belgian Trappist beers.