It would be good to have a solution for this.

Since the query language itself is not expressive enough (not currently possible to query for correlated data in json arrays like experiences for example), in order to find relevant data, we'd need to run queries with strict requirements first, then re-run them if there aren't enough results with increasingly loosened conditions to get more.

However, this means that we'd keep getting the same records as conditions loosen too, which we'd be charged for, so this solution does not work either.

I understand that providing a list of IDs to exclude may also not be a solution, since that could potentially mean an ever-growing request that may eventually start failing as further pages would be expanded.

A proposed solution that would work for this particular use-case, is if there was a concept of "query ID", either one that could be provided by a user, or returned in the first response by the API.

Then, if I use cursors to expand further pages using cursors, or if I were to loosen the query conditions, I'd also pass in the query ID, which would tell the API not to return results that were already returned in the context of this query.

The above would not guarantee that one would never receive repeat records over all, but it would allow us to at least run a search iteratively that would only result in unique records.

While this doesn't necessarily hit the exact ask in the post, it's important to point out that we are currently in the process of building out a new API called the Person Retrieve API. The end state of this product will allow you to send a persistent PDL ID to the endpoint along with a field of interest. If that field has yet to change since the last time you requested that profile, you will not be charged. You can see here for additional information: https://docs.peopledatalabs.com/docs/person-retrieve-api

To be clear, we are in the early rollout phase of this product. As of 3/14/22, you will be charged against this API regardless of the changes to the profile. We will be sure to communicate new features and functionality as they are made available.