VBA - scraping websites videos | Excel VBA Part 49 - Downloading Files from Websites

Posted by Andrew Gould on 21 November 2016

Excel VBA doesn't have a native method for downloading files from websites but you can declare an API function that will enable you to do this. This video takes you through the process of declaring the API function and using it in your code, along with a bunch of other useful techniques such as using folder pickers, creating folders with FileSystemObjects and opening a Windows Explorer window using the Shell function.

You can download any files that you need to follow the video here.

You can increase the size of the video:

Full screen mode for YouTube

You can view the video in full screen mode as shown on the left, using the icon at the bottom right of the frame.

You can also increase the quality of the video:

Changing resolution

You can improve the resolution of the video using another icon at the bottom right of the frame. This will slow down the connection speed, but increase the display and sound quality. This icon only becomes visible when you start playing the video.

Finally, if nothing happens when you play the video, check that you're not using IE in compatibility view.

This page has 4 threads Add post
18 Aug 18 at 22:32

 Andrew,

Thanks again for your awesome videos. 

On VBA part 47 video, you show us how to input a value, click the "Go" search button, and scrape contents from queried URL.  I used the same technique to scrape the page after the search button is clicked.  It didn't return value (innertext).

Wonder if you can help me out.

Thanks,

Suntao

20 Aug 18 at 07:27

You can reference the Document property of the IE object after navigating to the second page to access the HTML and text of the page.  You may need to introduce a delay to your code to allow IE to catch up with the instruction to navigate to a new page before you attempt to read its contents.  The example below shows how you could do this:

Sub NavigateToAnotherPage()

    Dim IE As New SHDocVw.InternetExplorer
    Dim HTMLDoc As MSHTML.HTMLDocument
    Dim HTMLParagraph As MSHTML.IHTMLElement
    Dim HTMLParagraphs As MSHTML.IHTMLElementCollection
    
    IE.Visible = True
    IE.navigate "en.wikipedia.org/wiki/Main_Page"
    
    Do While IE.ReadyState <> READYSTATE_COMPLETE
    Loop
    
    Debug.Print "First Page = ", IE.LocationName, IE.LocationURL
    
    IE.Document.forms("searchform").elements("search").Value = "Document Object Model"
    IE.Document.forms("searchform").elements("go").Click
    
    Do While IE.ReadyState <> READYSTATE_COMPLETE
    Loop
    
    'Wait two seconds
    Application.Wait Now + TimeValue("00:00:02")
    
    Set HTMLDoc = IE.Document
    Set HTMLParagraphs = HTMLDoc.getElementsByTagName("p")
    
    Debug.Print "Second Page = ", IE.LocationName, IE.LocationURL
    
    For Each HTMLParagraph In HTMLParagraphs
        Debug.Print HTMLParagraph.innerText
    Next HTMLParagraph
    
End Sub

12 Jun 18 at 14:37

Thank you sir for your tutorials

 please help me, i'm trying to run your example code, but it gives me error "access denied",

how can I solve this kind of error?

13 Jun 18 at 07:01

Hi, the most likely reason for the error is that the Wise Owl website has been updated to use https rather than http since the video was made.  You should be able to replace any reference to "http" in the code to "https" to solve the problem.

I hope that helps!

25 May 18 at 09:17

Viewed Video #47 on youtube.

Any plans to expand on XHR in Excel VBA, especially with session cookies?

25 May 18 at 12:34

There aren't, I'm afraid!

25 May 18 at 15:35

Please could you point in the direction of any instructional material or course or video that details only XHR and session cookies within an Excel VBA environment?

Andy B  
28 May 18 at 14:25

'Fraid i don't know of any, beyond Googling!

17 May 18 at 10:29

Hi Andrew,

Thank you so much for the 3 videos about Web scraping. It did guide me a lot on how to identify HTML code. Your approach is great. I seem unable to find any routine of auto login (username, password) to web site via VBA code.

Greetings,

Dude6571

18 May 18 at 07:53

Hi, we don't have a video on how to do this but there are plenty of suggestions out there for ways to do this:

https://stackoverflow.com/questions/48919491/login-to-a-website-using-vba

https://stackoverflow.com/questions/21244051/using-vba-to-automate-ie-login-having-errors

https://www.ozgrid.com/forum/forum/help-forums/excel-general/69057-automate-username-password-log-in-to-web-page

https://stackoverflow.com/questions/24038230/fill-user-name-and-password-in-a-webpage-using-vba

Of course the exact technique you'll use will depend on the page you're using to login but hopefully that's enough to point you in the right direction!