VBA HTML Navigate Through Listings

I've got the following code that WORKS, and pulls all of the links for listings in the webpage below. I am now looking to expand this to pull the next page of results (up to n).

Solution 1:

I would probably look to add in a test to ensure the number of pages you request are not greater than the available. Modularize the code a little to pull out the info extraction step. Use arrays and some basic optimization (Screenupdating) to speed up the whole process. Also, get rid of the ie object asap.

This with the listings results count set to 200 (which in fact gives 211 results per page with the given selector). Not sure if this is simply an ebay setting that is remembered or is default.

Option Explicit
Public Sub GetInfo()
    Dim ie As InternetExplorer, nodeList As Object, page As Long, totalResults As Long, ws As Worksheet
    Const RESULTS_PER_PAGE = 211
    Const DESIRED_PAGES = 3
    Const BASE = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_ipg=200&_pgn="
    Dim results(), url As String, maxPages As Long
    ReDim results(1 To DESIRED_PAGES)
    Application.ScreenUpdating = False
    Set ie = New InternetExplorer
    Set ws = ThisWorkbook.Worksheets("Sheet1")
    With ie
        .Visible = True
        For page = 1 To DESIRED_PAGES
            url = BASE & page
            .Navigate2 url
            While .Busy Or .readyState < 4: DoEvents: Wend
            If page = 1 Then
                totalResults = Replace$(.document.querySelector(".srp-controls__count-heading").innerText, " results", vbNullString)
                maxPages = totalResults / RESULTS_PER_PAGE
            End If
            Set nodeList = .document.querySelectorAll("#srp-river-results .s-item__link[href]")
            results(page) = GetLinks(nodeList)
            Set nodeList = Nothing
            If page + 1 >= maxPages Then Exit For
        Next
        .Quit
    End With
    If maxPages < DESIRED_PAGES Then ReDim Preserve results(1 To maxPages)
    For page = LBound(results) To UBound(results)
        If page = 1 Then
            ws.Cells(1, 1).Resize(UBound(results(page), 1)) = Application.Transpose(results(page))
        Else
            ws.Cells(GetLastRow(ws, 1) + 1, 1).Resize(UBound(results(page), 1)) = Application.Transpose(results(page))
        End If
    Next
    Application.ScreenUpdating = True
End Sub

Public Function GetLinks(ByVal nodeList As Object) As Variant
    Dim results(), i As Long
    ReDim results(1 To nodeList.Length)
    For i = 0 To nodeList.Length - 1
        results(i + 1) = nodeList.item(i)
    Next
    GetLinks = results
End Function

Public Function GetLastRow(ByVal ws As Worksheet, Optional ByVal columnNumber As Long = 1) As Long
    With ws
        GetLastRow = .Cells(.Rows.Count, columnNumber).End(xlUp).Row
    End With
End Function

Solution 2:

Untested (and I might be missing something), but seems like you can just specify the page you want to access with URL query parameter _pgn.

So for example, navigating to the URL below:

https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_ipg=200&_pgn=2

means you're requesting page 2 (also, the _ipg parameter seems to dictate how many results are shown on a page, so increasing to 200 might mean you need to make fewer requests overall).

So if you create some variable pageNumber in your code and increment it inside some sort of loop (that terminates once you've reached the last page), you should be able to get all pages -- or even any page at some arbitrary index --without copy-pasting/repeating yourself in your code.