Any research undertaking in the social sciences must confront the following questions: What is the evidence on which you base your conclusions? How reliable is that evidence? These questions permeated the papers and discussions at the workshop. As has been noted, speakers and participants concluded that the existing datasets all have limitations that inhibit their use. Some of these limitations render the data almost meaningless for some questions; others require creative work-arounds. The major issues include the following:
- The large, national level data sets do not provide data at a suitably granular level of detail. They support broad generalizations at the national level but they are not wellsuited for:
– Studying closely the behavior of firms;
– Refining models of workforce participation, incentives, and rewards;
– Understanding user adoption at local or regional levels. - Existing data sources data provide limited insight into the extent of consumer choice of broadband service providers. They also overstate the availability of broadband services because of the way in which they are reported.
- The definition of broadband articulated by the FCC is out of date and should be revised to capture current conditions. Accompanying that effort, the FCC should collect data in such a way so that researchers can develop an adequate picture of how the speed of broadband services is evolving.
- Zip codes are inconsistently defined by the two major agencies, the FCC and the Census Bureau, inhibiting merging datasets from multiple sources, necessary to obtain richer analysis. Other units of analysis, for example, the MSA and the school district, suffer from similar inconsistencies and misunderstandings. Solutions have been proposed to work around the inconsistencies, but nonetheless there remain severe limitations on what questions the data can support.
- Data measurements and even appropriate metrics to measure the quality of service experienced by end-users are lacking.
- Data on broadband pricing and business and household expenditures on broadband services are not readily available except in aggregate. Even here, meaningful categories of price data are not available in useful form.
- Data on broadband traffic patterns and its evolution over time are also not readily available. The appropriate metrics for analyzing such traffic and addressing such questions as the patterns of internet interconnections and routing policies are not welldefined, even within the technical community.
- Data that may be available from the private sector for business or residential users can be expensive, opaque, and even erroneous. The data provider may place restrictions of use of the data, as evidenced by the two major providers of clickstream data, comScore and Nielsen//NetRatings. Moreover, there is no guarantee that the owner will archive the digital data properly so that they are available for long term use.
- Privately funded academic research does compensate for some of the deficiencies in the large datasets, but these studies tend to be customized, focused and hence not necessarily comparable. Here, too, the data are not necessarily archived. Researchers have displayed varying levels of reluctance to deposit and reuse data which also can inhibit comparative studies as well as verify the analysis.”27
A recurring theme among researchers participating in the workshop is the importance of the local. Broadband connections are generally localized in a specific geographic area (i.e., wired connections are tethered to a specific location and wireless services are generally short-range). Significant differences in the availability and quality of service may occur over relatively short geographic distances, issues which are important to a full understanding of broadband markets. Moreover, the notion of quality of service might well be broadened beyond traditional metric of bandwidth to include assessment of the users’ experience to allow addressing such questions as “Are there measurable effects of improvements in user interface design?”28 or “Does security mean anything to end users and with the rise of identify theft, are they willing to pay for it?” With workshop participants repeatedly returning to the user and the context of use, a critical need is to collect data on broadband that are geographically disaggregated in order to understand the local impacts of broadband.
As the internet matures beyond a digital communications system to an infrastructure that supports a range of applications and services, we can expect the problems of measurement to become even more complex and difficult. At the same time that more granular data is needed, the landscape is becoming more intricate. More competitors are entering the market, offering different combinations of services for both infrastructure providers and end-users.
Finally, the workshop called for multidisciplinary and interdisciplinary approaches, engaging engineers who understand implications of changing technologies for users, economists with a broad variety of expertise, sociologists versed a range of skills from language to statistics, and industry participants with access to important datasets and a framework that enables them to share the information to advance the state of knowledge without incurring liabilities.
