- VIDEOS HOME PAGE
- .NET (14)
- Business Intelligence (40)
- Integration Services (19)
- Macros and Programming (82)
- Microsoft Excel (70)
- Microsoft Office (92)
- Miscellaneous (1)
- Power BI (35)
- Power Platform (35)
- Python (31)
- Report Builder (107)
- Reporting Services (113)
- SQL (42)
- Visual Basic for Applications (215)
- Visual C# (14)
- Excel VBA - Basics (24)
- VBA User Forms (22)
- Excel VBA - pivot tables (9)
- Excel VBA - charts (6)
- VBA - advanced (14)
- VBA - working with files (12)
- VBA - linking applications (12)
- VBA - working with Outlook (14)
- Built-in VBA functions (9)
- VBA - working with data (57)
- VBA - scraping websites (25)
- VBA - working with shapes (5)
- VBA - classes and structures (6)
VBA - SCRAPING WEBSITES VIDEOS▼
- Excel VBA Part 46 - Querying Web Pages with Query Tables
- Excel VBA Part 47.1 - Browsing to Websites and Scraping Web Page
- Excel VBA Part 47.2 - Scraping Website Tables and Clicking Links
- Excel VBA Part 47.3 - Internet Explorer vs XML HTTP Request
- Excel VBA Part 47.4 - Logging in to a Website with Windows Security
- Excel VBA Part 47.5 - Basic HTTP GET and POST Requests
- Excel VBA Part 48 - Scraping Multiple Web Pages
- Excel VBA Part 49 - Downloading Files from Websites
- Excel VBA Part 57.1 - Getting Started with Selenium Basic and Google Chrome
- Excel VBA Part 57.2 - Basic Web Scraping with Selenium and Google Chrome
- Excel VBA Part 57.3 - Using Different Web Browsers in Selenium
- Excel VBA Part 57.4 - Finding Web Elements in Selenium
- Excel VBA Part 57.5 - Implicit and Explicit Waits in Selenium
- Excel VBA Part 57.6 - Working with Multiple Tabs in Selenium
- Excel VBA Part 57.7 - Using Select Drop Down Lists in Selenium
- Excel VBA Part 57.8 - Printing in Google Chrome using Selenium
- How do I open Google Chrome in full screen mode using Selenium in VBA?
- Is it possible to start Google Chrome hidden using Selenium in VBA?
- How do I access elements in the Shadow DOM using Selenium in VBA?
- How do I Refer to a Parent Element in Selenium for VBA?
- How do I hold Shift and click a link in Selenium for VBA?
- How do I scroll down a web page using Selenium in VBA?
- How do I copy CSV data from a website using VBA?
- How do I open Google Chrome with a user profile using VBA?
- How do I use Google Translate with Excel VBA?
VBA - scraping websites videos | Excel VBA Part 47.5 - Basic HTTP GET and POST Requests
Posted by Andrew Gould on 05 June 2019
This video explains the basic differences between the GET and POST methods and how you can use each method to send data along with an HTTP request. You'll learn about adding a query string to a URL for GET requests and how to send values in the body of a POST request. You'll also see how to ensure the values you pass to a request are properly encoded using the Excel EncodeURL function. Throughout the video you'll also learn the basics of using the Chrome Developer Tools to work out how requests are sent by the Chrome browser which helps when translating this to VBA code.
This video has the following accompanying files:
|Basic HTTP GET and POST requests COMPLETED.xlsm||Excel workbook with macros|
|Basic HTTP GET and POST requests.xlsm||Excel workbook with macros|
Click to download a zipped copy of the above files.
There are no exercises for this video.
Making a video bigger
You can increase the size of your video to make it fill the screen like this:
Play your video (the icons shown won't appear until you do), then click on the full screen icon which appears as shown at its bottom right-hand corner.
When you've finished viewing a video in full screen mode, just press the Esc key to return to normal view.
Improving the quality of a video
To improve the quality of a video, first click on the Settings icon:
Make sure you're playing your video so that the icons shown appear, then click on this gear icon at the bottom right-hand corner.
Choose to change the video quality:
Click on Quality as shown to bring up the submenu.
The higher the number you choose, the better will be your video quality (but the slower the connection speed):
Don't choose the HD option unless you have a fast enough connection speed to support it!
Is your Wise Owl speaking too slowly (or too quickly)? You can also use the Settings menu above to change your playback speed.
Hello again and let me ask another question.
I wrote some function to search in header text but I have trouble to find destination url when I use MSXML and search an item in a website search form and that search guides me to a new url.
Is there any way to obtain the destination address? I really need that.
Another side issue is that using html with MSXML doesn't allow us to use two or more getelementby... in one line of the code. for example when I use IE I can write following code:
Set HTMLDiv = HTMLDoc1.getElementsByClassName("Class1")(0).getElementsByClassName(" Class2")(0)
but when using MSXML I have to break it into the following lines:
HTMLDoc2.body.innerHTML = HTMLDoc1.getElementsByClassName("Class1")(0).innerHTML
Set HTMLDiv = HTMLDoc2.getElementsByClassName(" Class2")(0)
Is there a simpler way to write it?
It sounds as though you're attempting to make use of features of the web page as provided by a web browser - unfortunately an XMLHTTP request doesn't provide that. If you need to interact with search forms and other parts of the web page I'd recommend automating a browser, either IE or Chrome using Selenium (we have several videos on using Selenium to automate Chrome).
I hope it helps!
Hello, thank you for your lessons that I can say are the best among all of the relative tutorials I have seen.
I have a problem with parsing data from the "HEAD" part of html.
by using "HTMLDoc.body.innerHTML = XMLRequest.responseText" we lose this part of html that I exactly need to scrape data from.
I searched the whole internet but I couldn't find the solution.
I need to extract a lot of data from this part so I can't use the InStr function and ...
I would be very grateful if you could help me with this.
I'm trying to use this code to login to a website.
Sub GetTable2() Dim req As New MSXML2.XMLHTTP60 Dim url As String url = "https://1305081.app.netsuite.com/app/common/search/searchresults.nl?searchid=1704&whence=" req.Open "Post", url, False req.setRequestHeader "Content-Type", "application/x-www-form-urlencoded" req.send "email=(myemail)&password=(mypassword)" 'check if page loaded If req.Status <> 200 Then MsgBox req.Status & " - " & req.statusText Exit Sub End If saveHTMFile req.responseText End Sub
When I get to the point of 'check if page loaded, the message box says "Internal Server Error." Any idea what could be wrong?
Thank you for your very helpful tutorial entitled "Excel VBA Introduction Part 47.5 - Basic HTTP GET and POST Requests." I have run into a problem that stumps me. Following the tutorial, I get the following result:
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no">
<meta name="mobile-web-app-capable" content="yes">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,500,600,700,800">
<link rel="apple-touch-icon" sizes="60x60" href="/assets/images/favicons/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/assets/images/favicons/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/assets/images/favicons/favicon-16x16.png">
<link rel="manifest" href="/assets/images/favicons/site.webmanifest">
<link rel="mask-icon" href="/assets/images/favicons/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
<link href="app.12d5769f220fe4fa467b.css" rel="stylesheet"></head>
However, when I inspect the elements of the website, I get the information I am looking for. I believe the problem is that the java script is not running, and I have been unable thus far to find out how to make the java script run so that I can scrape the portions of it that interest me.
If you are able to give me a suggestion or a work around, I would be very grateful.