Compared to the datasets you get access to in industry, most public datasets are tiny. It is like pulling teeth to get any large research, government, or industry datasets released so that they are publicly redistributable.

Some of the concerns I’ve run into:

* copyright (web crawls, image datasets, audio, product metadata)
* privacy concerns (collaborative filtering, government data, geo data, legal/pacer)
* competitive reasons (financial data, retail data, commerce, website stats)

I think we need a data stimulus plan :) Like the Enron email dataset on a larger scale, we should have open access to factual data from the financial markets, geodata, bankrupt companies, and other locked down data that the government has ties to. Pumping that government subsidized data out on the net could spur a lot of innovation which is something we need right now.