Who Owns the Open Web? Data Infrastructure, AI, and the Politics of Access

Well, let's talk a little bit more about that question of privacy and transparency, because one of the kind of, I suppose, perversities that I notice in the industry is that, to turn back to the lawsuit and what you elucidated about X and about Meta in the context of that lawsuit, is that some of these institutions that are most hostile to scraping are also institutions that have become incredibly resistant to external accountability.

And there's a link, I think, between transparency, allowing data to be used for research and understanding and that kind of accountability.

And researchers and journalists and civil society groups often turn to scraping data.

Because the platforms themselves are so resistant to being analyzed and to providing researchers the data that we need to make any kind of argument or case about regulation, for example, they no longer provide meaningful visibility into what's going on in the platforms.

And so web scraping becomes the kind of last and only line.

of accessibility to understanding what is happening on those platforms in a meaningful way that scholarship and

research can

be served by.

Do you think that that's fair?

So absolutely not.

And the way you looked at it is exactly the way we see the industry.

Obviously, each and every one of us is a person or his organization would love to just go in the main path, right? Come into whatever website and ask them to get access via an API, even if you need to pay something because they have their own operation and storage costs in order to be able to use the data for any type of research or development you want to do.

And we even offer these big players a collaboration path where we will help them understand when Deb is coming as a researcher for Berkeley, you probably don't want to prevent access to her because you understand that the work that she's going to be doing is very critical and can't be done without access to that specific data set.

Having said that, most big organizations, and by the way, that includes corporate, but also government, right, and many other kind of core holder organizations shareholder of data, are actually doing exactly the opposite, as you described.

And because we believe that if you want to use the data, you should be able to use it as long as it's public, then absolutely scraping solutions are becoming the go-to, and that makes not only researchers, but not only big companies, but also researchers or nonprofit, which we might, if we have time, happy to kind of talk a little bit about that program that we're running, need to use the data to do like not only commercial solution, but like literally life-saving solution that I think all of us could not say anything against it and would believe that the access should remain.