Marketers, media companies, agencies, and researchers frequently come to us with questions about how to make sense of the enormous data streams at their disposal. While we’re accustomed to doling out advice and pointing people to relevant studies on set-top-box data, scanner data, or clickstream data, we want to challenge the industry to think bigger when it comes to big data. So we went as big as we could—outer space. We caught up with Jon Jenkins, Senior Research Scientist, SETI Institute at NASA Ames Research Center, and asked him how he makes actionable insights out of galaxies of data. We hope his answers will give you some perspective on how to deal with your own data issues, even if they’re merely terrestrial.

ARF: Can you briefly describe the research you do?

Jenkins: I develop science algorithms for processing photometric data from the Kepler mission to produce science archive products and to detect and characterize weak planetary signatures in the data. Our goal is to determine what fraction of stars in our galaxy host potentially habitable Earth-size planets. Kepler takes images of over 150,000 stars in order to detect small dips in the brightness caused by instances where a rocky planet like Earth passes in between the space telescope and the planet’s host star. We have to process the raw image data to calibrate it and extract brightness measurements for each star for each half hour interval. The pipeline corrects the data for instrumental effects and then searches for these very weak signatures in the data to identify stars that might have planets. The pipeline then conducts a series of diagnostic tests to make or break the confidence in the signature’s planetary nature. We furnish the list of stars with transit-like features and the diagnostics to the science team for follow up and analysis.

ARF: What are the biggest challenges you face in making sense of such large data sets?

Jenkins: The biggest challenge is to learn how various instrumental effects manifest themselves across the 95 million pixels in Kepler’s camera and across the ~200,000 stars we’ve observed since the start of the mission, each of which has its own unique behavior to consider.

Kepler is NASA’s first mission capable of finding Earth-size planets orbiting Sun-like stars. It’s an order of magnitude better than any previous space photometer and about 2 orders of magnitude better than any ground-based photometer. Kepler is collecting data with unprecedented precision, duration and contiguity. The photometer is exquisitely sensitive, otherwise it couldn’t do its job, but it’s also sensitive to its thermal state. We’re finding that the biggest challenge in finding the very small signals we’re looking for is caused by a combination of instrumental effects and the fact that most of our target stars appear to be more variable with respect to their brightness output than our Sun. Dealing with large data sets has forced us to design a set of processing clusters that can keep up with the data accumulation rate. However, we have moved the most computationally intensive parts of the pipeline, namely the identification and characterization of planetary signatures, to the Pleiades supercomputer at NASA Ames Research Center (which is also where the Kepler Science Operations Center is located).

We are learning about how to deal with the instrumental effects as we gain experience with the data and the spacecraft, and it is frustrating that we can’t instantaneously reprocess all the data as we upgrade the pipeline. I wish we could reprocess the data much faster than we can, as it will take us about 7 months to reprocess all the data we currently have and “catch” up with the new data with the new software we’re just about to release. We are limited by the speed with which we can pull data from the filestore and the number of processing cores we have available with our own clusters.

ARF: What kinds of general questions should researchers be asking when working with big data? Can you give any advice on how to conceptualize enormous datasets like those you work with?

Jenkins: I think it’s very important to think beyond the use case of the nominal processing when you are designing an automated pipeline that must process significant amounts of data, especially if that data set accumulates over time and full reprocessing is a necessity. I’m thinking specifically about the use case of accessing the data to diagnose problems with the software or with the instrument and the fact that the scientists and programmers who are evolving the software need to be able to access and work with large fractions of the data in order to do a credible job. If you design a pipeline to run well in an unsupervised environment and don’t think about how the people responsible for the quality of the data products need to interact with the data and how they need to prototype and test new algorithms, you may unintentionally hamstring your ability to move forward in solving the unknown problems that inevitably happen when you are breaking new ground in science and technology.

John Jenkins spoke at our Industry Leader Forum on October 27, 2011. This post originally appeared on our MediaBizBloggers Site.

Set-top-box data, loyalty card data, cookie data, and more have created vast databases of customer data. While this gives marketers great opportunities to reach receptive customers, making sense of big data can be a challenge. We caught up with John LaRocca, Vice president, Strategic Partnerships at dunnhumbyUSA to get his expert opinion on making business sense of massive data sets.

ARF: What are the most exciting new data sources you’ve been able to use in the last decade?

John LaRocca: As someone who has spent a career analyzing and gleaning insights from syndicated panel and point-of-sale databases and shopper loyalty card data, today is a tremendous opportunity to better understand customers in ways we never thought possible. Although much of the data is anonymous, this type of data allows us to understand the behavior and motivations of a larger universe of customers. It gives us the resources to personalize an experience and deliver value through relevance in areas like assortment, pricing and promotions, for example. It gives us the tools identify customer growth and attrition and empowers brands and retailers to develop better relationships with their best customers.

There is an effort today to associate these databases with customer data from other sources, to develop a better understanding of the customer on the path to purchase. These efforts are revolutionizing the advertising industry, for example, where ROI and effectiveness has traditionally been difficult to measure and targeting has largely been based on panel-based demographics. At dunnhyumbyUSA, I am responsible for building collaborative partnerships with other data and solutions providers to create new solutions and enable our clients to deliver more value to customers in impactful ways. We have a collaborative agreement with comScore to measure the effectiveness of online advertising, pairing online and in-store behavior. We have a similar agreement with TRA, which measures the television viewing habits of two million set-top box households, to develop a similar understanding of the effectiveness of television advertising planning that is also based on actual purchases.

Integrating a customer-driven approach across channels, connecting in-store with online, for example, begins to shed light on a shopper’s decision process throughout the path to purchase. And I think we are only beginning to understand its potential.

ARF: What have we learned about customers from big data?

JL: We have certainly learned a lot of new things about the customer, but the data and resulting insights have also confirmed, quantified, and expanded some existing beliefs:

• Each customer is unique and, therefore, brands need to treat them as such, including adopting a personalized approach in how they communicate with them, rather than at them.

• Customers behave differently in different categories. A “price sensitive” customer is not equally price sensitive in every category. A “discriminating” customer is not consistently discriminating across the store.

• People don’t behave as they claim, and that actual behavior is a better starting point for understanding WHY customers do what they do than merely asking them. And when you link actual behavior with attitudes, you create insights that are game changing.

• Demographics, psychographics and attitudinal data are poor tools for targeting and interpreting customer preferences because they drive generalizations, reinforce the myth of the “average” shopper and do not correlate with behavior.

• Focusing on customer acquisition can be an expensive choice. At dunnhumby, we’ve found that it takes between 12 to 20 new customers to make up for the loss of just one committed customer. I think we will find out that this number increases dramatically as we gain a greater understanding of social networks.

ARF: What are some of the nagging problems you don’t think we can address because they’re too time-consuming, costly, or require too much processing power? Do you see the industry ever solving these problems?

JL: Many of the challenges companies face are due to both the quantity and complexity of the data available. Most companies collect data but struggle to understand how to use the data they collect effectively to drive business results and leverage data-driven insights throughout their organizations. Data itself is in danger of becoming a commodity. Any company can collect data, but the real value comes from leveraging that data to create even more valuable data (e.g., segments, dimensions), develop the right insights and discover the “customer /DNA.”

As databases continue to grow, real-time reporting is a challenge. There are more data dimensions to measure across multiple products, geographies, time periods, and customer segments. For example, online shopping has created this long tail of UPCs because there is no limit to the number that can be “shelved” in the digital store. Thus, the number of items in the report has grown significantly and will continue to do so.

It isn’t only the explosion of raw data, but the complexity it drives in the data cube that creates the reporting challenge. There are more disparate sources of customer behavioral data — from loyalty cards, search/browsing, social, mobile, and set-top-box data, among others – each with aggregation and manipulation challenges, that have created opportunities to generate an increasing number of behavior-based customer segments to report, analyze, and model. This complexity creates an issue with the timeliness of insights. People have become accustomed to simplified, instantaneous results and so, when they have to wait an hour to two, or overnight, it creates a lot of frustration.

To put it simply, we have the database management and business intelligence tools to find the needle in the haystack but the haystack is now bigger. I don’t believe that these issues are too great to solve. The industry typically finds a creative way to solve its problems.

John LaRocca is Vice President, Strategic Partnerships at dunnhumbyUSA responsible for developing the company’s partnership strategy within media and analytics, identifying and leveraging collaborative partnerships with companies that have complementary data assets and capabilities in order to develop innovative, customer-driven solutions for dunnhumby clients. He plays a critical role in enhancing dunnhumby’s media measurement solutions and works to expand the company’s capabilities in this area.

This post originally appeared on our MediaBizBlogger site.