VBA - scraping websites videos | Excel VBA Part 47.5 - Basic HTTP GET and POST Requests

Posted by Andrew Gould on 05 June 2019

This video explains the basic differences between the GET and POST methods and how you can use each method to send data along with an HTTP request. You'll learn about adding a query string to a URL for GET requests and how to send values in the body of a POST request. You'll also see how to ensure the values you pass to a request are properly encoded using the Excel EncodeURL function. Throughout the video you'll also learn the basics of using the Chrome Developer Tools to work out how requests are sent by the Chrome browser which helps when translating this to VBA code.

This video has the following accompanying files:

File name Type Description
Basic HTTP GET and POST requests COMPLETED.xlsm Excel workbook with macros
Basic HTTP GET and POST requests.xlsm Excel workbook with macros

Click to download a zipped copy of the above files.

Making a  video bigger

You can increase the size of your video to make it fill the screen like this:

View full screen

Play your video (the icons shown won't appear until you do), then click on the full screen icon which appears as shown at its bottom right-hand corner.

 

When you've finished viewing a video in full screen mode, just press the Esc key to return to normal view.

Improving the quality of a video

To improve the quality of a video, first click on the Settings icon:

Settings icon

Make sure you're playing your video so that the icons shown appear, then click on this gear icon at the bottom right-hand corner.

 

Choose to change the video quality:

Video quality

Click on Quality as shown to bring up the submenu.

 

The higher the number you choose, the better will be your video quality (but the slower the connection speed):

Connection speed

Don't choose the HD option unless you have a fast enough connection speed to support it!

 

Is your Wise Owl speaking too slowly (or too quickly)?  You can also use the Settings menu above to change your playback speed.

This page has 1 thread Add post
09 Jul 20 at 01:50

Thank you for your very helpful tutorial entitled "Excel VBA Introduction Part 47.5 - Basic HTTP GET and POST Requests."  I have run into a problem that stumps me.  Following the tutorial, I get the following result:

***************************
<!doctype html>
<html lang="en">
<head>
  <base href="/">
  <meta charset="UTF-8">
    
    <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no">
  <meta name="mobile-web-app-capable" content="yes">
  <title>Profits Run</title>
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,500,600,700,800">

  <link rel="apple-touch-icon" sizes="60x60" href="/assets/images/favicons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/assets/images/favicons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/assets/images/favicons/favicon-16x16.png">
  <link rel="manifest" href="/assets/images/favicons/site.webmanifest">
  <link rel="mask-icon" href="/assets/images/favicons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  
<link href="app.12d5769f220fe4fa467b.css" rel="stylesheet"></head>
<body>
  <div id="root"></div>
<script type="text/javascript" src="vendor-d7a24874.js"></script><script type="text/javascript" src="app-4c8acbc2.js"></script></body>
</html>
***************************
However, when I inspect the elements of the website, I get the information I am looking for.  I believe the problem is that the java script is not running, and I have been unable thus far to find out how to make the java script run so that I can scrape the portions of it that interest me.  

If you are able to give me a suggestion or a work around, I would be very grateful.  

Thank you

09 Jul 20 at 09:14

Hi, the easiest way to get the javascript to run is to use VBA to open the page in a web browser.  We have a few videos on how to do this using Internet Explorer (which you may have already seen).  If you’re interested in using Chrome you might like to investigate the Selenium library.  We don’t have any resources on Selenium but you can find a basic tutorial here. I hope that helps!