You can absolutely download the data in bulk, as long as you don’t try to do that every 5 mins. Regarding crawling we mean repeatedly calling the API to try to gradually build your own data set - which is completely unnecessary, error-prone, a bit dumb, and just costs us CPU.
To download the global data set you need an API call like the following (for example using curl or another utility to save the results to a file, because the global data set will be very large and likely more than your web browser can cope with).
This example will return up to 50 results (globally):
This next example will return up to 50 results (globally), with whitespace and nulls removed (
compact=true) and extended reference data properties removed (using
verbose=false, so things such as DataProvider where you’ll just get our ID for them instead):
So to download the entire data set, you’d set
maxresults=200000 (for instance).
You can optionally get results as CSV file (with
output=csv) if you don’t want a JSON response.
Let me know if you have any more questions.