Web Scraping with Python, Ruby & Scrapy | Charlie Craine | Skillshare

Web Scraping with Python, Ruby & Scrapy

Charlie Craine

Play Speed
  • 0.5x
  • 1x (Normal)
  • 1.25x
  • 1.5x
  • 2x
23 Videos (2h 45m)
    • Web Scraping Intro

    • Installing Ruby

    • Walmart: How to scrape all pages from a product category on Walmart.com

    • Spoof a site and look like you are web browser!

    • Google + Mechanize = How to scrape search results from Google with Mechanize

    • Scraping YouTube's Most Popular

    • Scraping Zillow by Zip Code

    • Using JSON to scrape Walmart's website

    • Scrape a Tweets from another User or Users

    • Scrape Twitter searches for keywords and anyone who has tweeted using that keyw

    • Take the scraped users and follow them or swap out for another username

    • Installing and setting up Python

    • Simple Script: Learn how to crawl IMDB

    • Learn how to submit a web form

    • Install Selenium and PhantomJS - Trust me you'll want these!

    • Yes, you can crawl JavaScript! Learn to scrape Ajax with Selenium

    • Simple: Crawl and export data from Nordstrom.com into a CSV

    • Next Step: Crawl NFL data - Grab all the QB data from 2014!

    • Advanced: Export NFL QB data to a CSV

    • Export Data to MySQL

    • Installing and setting up Scrapy

    • Craigslist: Use Scrapy for a basic crawl of 100 jobs

    • Craiglist: Crawling through paginations for all jobs


About This Class

You'll be web scraping in less than an hour!!

This course teaches you three ways to scrape websites:

Import. io is a software that you download and is a really quick way to start web scraping quickly. It is not without its limitations but depending on what you need or plan to do it could do everything you need. No programming experience necessary!

If you prefer to learn a little programming along the way, and aren't afraid to invest a little time, you can learn how to do web scraping with Ruby and/or Python. Both have their pluses and minuses but both can do just about anything you could imagine.

Even if you have never used Python or Ruby you can learn how to still use these programs as a web scraper. But if you are willing to learn you can scrape just about any website there is online!

Want to do some web scraping of super popular websites? That's how you will learn!

This course comes complete with code for every lecture for Ruby and Python!

You will learn to web scrape popular websites like:

  • Zillow
  • Craigslist
  • Yelp
  • IMDB
  • Wikipedia
  • Google
  • Macys
  • YouTube
  • NFL
  • Nordstrom
  • Wal-Mart
  • Twitter

You will learn to install Python and Ruby. You'll learn how to download what you web scrape into CSV files and with Python you will learn how to put your data into a database.

Other lessons include how to get around obstacles such as websites with ajax that load content dynamically. You'll also learn how to crawl multiple pages of websites and get all the data you need!

With Import. io you will learn how to download a website that has over 40,000 products on it. Yes, 40,000 products and you will learn how to scrape all of those products.

Stop imagining you can grab data off websites and use them for your next web project, you an do it now.

Please note: Software installation can be done for both Mac OSX and Windows. I use a Mac so it is more focused on OSX however instructions are available for Windows.





  • --
  • Beginner
  • Intermediate
  • Advanced
  • All Levels
  • Beg/Int
  • Int/Adv


Community Generated

The level is determined by a majority opinion of students who have reviewed this class. The teacher's recommendation is shown until at least 5 student responses are collected.

Over 17 years of digital media experience. Currently, CTO executive for B2B publishing company with eight magazine websites as well as lead at its Interactive Agency working with Fortune 500 companies.

Our agency specializes in web development, marketing, SEO, social and email for B2B publisher. Leads agency web development, video, SEO & app development.

Previous experience includes Digital General Manager over cool magazines such as Guitar World and Revolver Magazine. And Gener...

See full profile

Report class