The vast majority of modern large websites today will have multiple backend APIs which connect to their frontend apps such as websites or mobile applications.
Generally speaking, if your target site has a backend API which contains all the data you require, you can interface with it directly. If possible, you'll save a ton of compute power - meaning your entire collection process will be cheaper: less bandwidth required, less intrusive (hence less likely to get blocked), and a smaller server will be required.
Running a headless browser is a fairly compute intensive task, especially if you are running multiple threads and wish to obtain data fairly quickly.
Hence, finding a hidden API is valuable and worth spending a bit of time trying to figure out.
Where is the Hidden API?
This can be done fairly simply by using the 'network' tab inside any web browsers developer toolkit:
Generally all you'll need to do now is reload the page (perhaps clearing cache as well) and ensure the network tab is open. Now watch all the various requests. Often filtering by 'XHR' will find an API endpoint, but be sure to check them all.
Be sure to check the 'response' section and eventually you might find something like this:
This could be JSON, XML or similar. Of course your web scraping project may require that you query different API endpoints to obtain the data that you require. Often you'll simply need to navigate the site like a normal user, keeping the network tab open, and step through each network request manually.
Once you've found the target endpoint, try right clicking and open in a new tab. If this fails to load, you will need to setup your scraper with the correct HTTP headers. This can be done fairly easily by right clicking, selecting 'Copy as Curl' and extracting the required headers. For example, many APIs will require that you send the correct 'Accept' header.