Vba Html Navigate Through Listings

I've got the following code that WORKS, and pulls all of the links for listings in the webpage below. I am now looking to expand this to pull the next page of results (up to n).

Solution 1:

I would probably look to add in a test to ensure the number of pages you request are not greater than the available. Modularize the code a little to pull out the info extraction step. Use arrays and some basic optimization (Screenupdating) to speed up the whole process. Also, get rid of the ie object asap.

This with the listings results count set to 200 (which in fact gives 211 results per page with the given selector). Not sure if this is simply an ebay setting that is remembered or is default.

Option Explicit
Public Sub GetInfo()
    Dim ie As InternetExplorer, nodeList AsObject, page As Long, totalResults As Long, ws As Worksheet
    Const RESULTS_PER_PAGE = 211Const DESIRED_PAGES = 3Const BASE = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_ipg=200&_pgn="
    Dim results(), url AsString, maxPages As Long
    ReDim results(1 To DESIRED_PAGES)
    Application.ScreenUpdating = False
    Set ie = New InternetExplorer
    Set ws = ThisWorkbook.Worksheets("Sheet1")
    With ie
        .Visible = TrueFor page = 1 To DESIRED_PAGES
            url = BASE & page
            .Navigate2 url
            While .Busy Or .readyState < 4: DoEvents: Wend
            If page = 1 Then
                totalResults = Replace$(.document.querySelector(".srp-controls__count-heading").innerText, " results", vbNullString)
                maxPages = totalResults / RESULTS_PER_PAGE
            End If
            Set nodeList = .document.querySelectorAll("#srp-river-results .s-item__link[href]")
            results(page) = GetLinks(nodeList)
            Set nodeList = Nothing
            If page + 1 >= maxPages Then ExitFor
        Next
        .Quit
    End With
    If maxPages < DESIRED_PAGES Then ReDim Preserve results(1 To maxPages)
    For page = LBound(results) To UBound(results)
        If page = 1 Then
            ws.Cells(1, 1).Resize(UBound(results(page), 1)) = Application.Transpose(results(page))
        Else
            ws.Cells(GetLastRow(ws, 1) + 1, 1).Resize(UBound(results(page), 1)) = Application.Transpose(results(page))
        End If
    Next
    Application.ScreenUpdating = True
End Sub

PublicFunctionGetLinks(ByVal nodeList AsObject) AsVariantDimresults(), iAsLongReDimresults(1 To nodeList.Length)
    Fori = 0 TonodeList.Length - 1
        results(i + 1) = nodeList.item(i)
    NextGetLinks = resultsEndFunctionPublicFunctionGetLastRow(ByVal ws As Worksheet, Optional ByVal columnNumber As Long = 1) AsLongWithwsGetLastRow = .Cells(.Rows.Count, columnNumber).End(xlUp).RowEndWithEndFunction

Solution 2:

Untested (and I might be missing something), but seems like you can just specify the page you want to access with URL query parameter _pgn.

So for example, navigating to the URL below:

https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_ipg=200&_pgn=2

means you're requesting page 2 (also, the _ipg parameter seems to dictate how many results are shown on a page, so increasing to 200 might mean you need to make fewer requests overall).

So if you create some variable pageNumber in your code and increment it inside some sort of loop (that terminates once you've reached the last page), you should be able to get all pages -- or even any page at some arbitrary index --without copy-pasting/repeating yourself in your code.