We are currently experiencing an explosion of "big data." Big data is "a collection of digital data sets so large and complex that it is beyond the capabilities of current database management tools," said Robert C. Johns, associate administrator and director of Volpe, The National Transportation Systems Center

Prominent thought leaders and experts from the federal government, academia, and the private sector recently came together at Volpe, The National Transportation Systems Center, to discuss challenges and opportunities related to big data.

The challenges for federal agencies, industry, and other organizations are to figure out how to manage and analyze big data to create value, increase accurate predictions, and provide approaches that drive sound decision making. As the Obama Administration announced in March 2012, the Big Data Research and Development Initiative—moving data to knowledge to action—has great potential to impact key national priorities, including transportation and defense.

With great challenges come new insights and opportunities—new technologies and processes are being considered, industry trends and best practices are being reviewed, and deep data mining is being discussed. A workforce with deep analytical experience will be needed and managers will need to be data literate. Those that learn how to unite domain expertise with data science will move to the forefront of their fields.

Over 35 participants joined the meeting on "Big Data Analytics: Driving Value Beyond the Hype," which was co-hosted by Volpe and the Hanscom Air Force (AF) Base Life Cycle Management Center (LCMC). Johns and AFLCMC Chief Architect and Chief Technology Officer Dr. Tim Rudolph were joined by Dr. Mark Linderman from Air Force Research Laboratory in Rome, NY; Anne Aylward, deputy associate administrator for Research and Innovation at Volpe; and other AF and Volpe officials. The Innovation Institute at the Massachusetts Tech Collaborative was also represented.

The meeting featured presentations from Johns; keynote speaker Dr. Anne Quaadgras, research scientist at Massachusetts Institute of Technology, Sloan School Center for Information Systems Research; Frank Stein, director of IBM's Analytics Solutions Center; and Peter Bostrom, transformation director, Department of Defense, Engineered Systems at Oracle Public Sector. Tim Schmidt, acting chief information officer for the U.S. Department of Transportation, led moderated discussions during this meeting.

Johns offered opening remarks and cited recent reports which indicate that in most organizations, the volume of data is expanding by 35 to 50 percent every year. Today's companies process more than 60 terabytes of information annually—about 1,000 times more than a decade ago, said Johns.

Now that 85 percent of data is unstructured and only 15 percent is structured, there is the significant challenge of how to process big data in real time. However, whether it is structured or unstructured data, the same questions arise: How are we going to retrieve the data, act on it, and store it? Retrieving data and getting value from data can be difficult. The key is creating cultural change in how an organization uses its data.

"Using data well means working smarter on a daily basis," said Dr. Quaadgras. "Working smarter is about using the 'little data,' ubiquitous in an organization's digital platforms, that can most easily be leveraged for success."

From his research, Stein noted, "Watson is a big data analytical system that can mine vast quantities of data to get one small piece of data that is wanted. The Watson system is an approach to use against big data." New big data brings opportunities and requires new analytics. There must be a change in the paradigm for big data to work. By moving analytics to the data, this changes technology.

There are a number of programs and projects underway where agencies and organizations are building big data ecosystems. Bostrom outlined how to integrate these enormous amounts of small pieces of information in real time. "For example, persistent intelligence, surveillance, and reconnaissance is moving analytics to where the sensors are and determines what is important in order to solve too much data coming across a too narrow pathway for reaching the decision maker," said Bostrom.

During the moderated discussions, Schmidt asked many thought-provoking questions and began by asking, "What are the federal challenges we are facing today? Not so much that there is one silver bullet of best practices, but rather…Can we use data known to us to pinpoint who needs what? How do we find data sets and leverage them? How do we get big data down to manageable data?"

"With companies putting in enterprise resource planning, how do we use the sea of data coming out of ERPs?" said Dr. Quaadgras. "This is the next step: Get at the data and not enter another ERP into the pipeline. Make investments in the people, not the pipe. Coaching and live human connection is a key differentiator versus inadequate data. The focus is on this concept. Data and analytics are not great yet, but we can make it better."

There was significant engagement in these discussions, with several participants offering their expert opinions and vast knowledge on the subject of big data. Other responses pertained to common challenges and privacy issues to automating legacy systems and building applications that leverage data sets while lowering costs.

For further information, please contact Ellen Bell, Volpe's director of Strategic Initiatives for Research and Innovation, at 617-494-2491 or ellen.bell@dot.gov.

Photo above: Volpe and Hanscom Air Force Base co-host big data meeting at Volpe. (Volpe photo)