If you found this video useful and you’d like to say thanks you can click here to make a contribution. Thanks for watching our videos!

BY CATEGORY▼

VBA CATEGORIES▼

VBA - SCRAPING WEBSITES VIDEOS▼

VBA - scraping websites videos | Excel VBA Part 47.3 - Internet Explorer vs XML HTTP Request

Posted by Andrew Gould on 03 June 2019

This video shows you how to use VBA to scrape websites using Internet Explorer and XML HTTP Requests. You'll learn how to reference the correct object libraries, how to create an XML HTTP Request and capture the response text. You'll compare the performance of the XML HTTP Request with Internet Explorer by scraping a website of gambling odds and parsing an HTML table, writing the results to a new Excel worksheet.

You can download any files that you need to follow the video here.

You can increase the size of the video:

You can view the video in full screen mode as shown on the left, using the icon at the bottom right of the frame.

You can also increase the quality of the video:

You can improve the resolution of the video using another icon at the bottom right of the frame. This will slow down the connection speed, but increase the display and sound quality. This icon only becomes visible when you start playing the video.

Finally, if nothing happens when you play the video, check that you're not using IE in compatibility view.

This page has 2 threads Add post

M G Joshi Reply to M G Joshi Reply

09 Oct 19 at 21:21

Dear Sir -- when running the following code I get the VBA error
>>> Run Time error '91'
>>> Object variable or With block variable not set

Sub ScrapeOddsUsingXMLHTTP()

    Dim XMLRequest As New MSXML2.XMLHTTP60
    Dim HTMLDoc As MSHTML.HTMLDocument
    Dim HTMLDiv As MSHTML.IHTMLElement
    Dim HTMLTable As MSHTML.IHTMLElement

    XMLRequest.Open "GET", "https://www.oddschecker.com/golf/houston-open/winner", False
    XMLRequest.send

    If XMLRequest.Status <> 200 Then
        MsgBox XMLRequest.Status & " - " & XMLRequest.statusText
        Exit Sub
    End If

    HTMLDoc.body.innerHTML = XMLRequest.responseText   ' <== Code breaks down at this point

    Set HTMLDiv = HTMLDoc.getElementById("oddsTableContainer")
    Set HTMLTable = HTMLDiv.getElementsByTagName("table")(0)

    Debug.Print HTMLTable.className
End Sub

Any suggestions? Thank you.

Andrew G Reply to Andrew G Reply

11 Oct 19 at 10:26

Hi,

You haven't created a new instance of the HTMLDocument class before attempting to change its .body.innerHTML property.

You can do this either by adding this line to your code:

Set HTMLDoc = New MSHTML.HTMLDocument

Or by changing the variable declaration like so:

Dim HTMLDoc As New MSHTML.HTMLDocument

I hope that helps!

AceAxe Reply to AceAxe Reply

18 Jun 19 at 09:25

Dear Andrew,

Has anything changed in that web-site (https://www.oddschecker.com.....) since you pasted this video?

Because no matter how I try to run the macro using IE or HTTP Request result is the same (slightly different but).

IE gives me following messages:

Can’t reach this page

•Make sure the web address https://www.oddschecker.com is correct
•Search for this site on Bing
•Refresh the page

More information
Error Code: INET_E_DOWNLOAD_FAILURE

Can’t connect securely to this page

This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner.

Your TLS security settings aren’t set to the defaults, which could also be causing this error.

Try this:
•Go back to the last page

and HTTP Request shows me that funny msgbox:

Run-time error '-2146697208(800c0008)':

The download of the specified resource has failed.

I tryed to use my code that I wrote watching your lesson or when I ran your code that I downloaded from your web-site.

Is anything wrong with my machine or there's something else I do not understand.

Dear Andrew,

Thatk you so much

Sincerily yours

Alexander

Andrew G Reply to Andrew G Reply

18 Jun 19 at 13:10

Hi Alexander,

I'm not sure why you're receiving this error message but it seems that you're not the only person to experience it. There are some possible solutions listed in these forums

https://answers.microsoft.com/en-us/windows/forum/windows_10-networking/cant-connect-securely-to-this-page-site-uses/5f1e1e7c-dc02-434c-8ffb-83c3e6e8b512

https://www.tenforums.com/browsers-email/90236-microsoft-edge-tls-security.html

Hopefully one of those will help!

Add a new post