A multidisciplinary team led by ISI researcher Aram Galstyan is developing a scalable hybrid forecasting platform that incorporates human insights and machine learning models to predict socio-economic and geopolitical events. Photo/iStock.
A research team led by the Information Sciences Institute (ISI) at the University of Southern California’s Viterbi School of Engineering has received a four-year, multi-million dollar Intelligence Advanced Research Projects Agency (IARPA) grant to develop a human-machine hybrid forecasting system.
The goal is to produce more accurate forecasts of worldwide socio-economic and geopolitical issues, such as disease outbreaks, stock market fluctuations, elections, and interstate conflict. ISI’s research team, named SAGE (Synergistic Anticipation of Geopolitical Events), is led by Aram Galstyan, director of ISI’s Machine Intelligence and Data Science (MINDS) group. Collaborators include UC Irvine, Fordham University, Stanford University and Columbia University. The team is one of three groups selected by IARPA to take part in the multi-year program, named the Hybrid Forecasting Competition. HRL Laboratories and BBN Technologies Corporation will spearhead separate teams.
Hybrid human-machine system
The SAGE team, comprising computer scientists, engineers and social and cognitive psychologists, will develop a scalable, hybrid forecasting platform that incorporates human insights and machine learning models. This hybrid human-machine system will allow data-driven platforms to incorporate human feedback “on the fly” for emerging issues that lack historical precedent.
“Fundamentally, this program is about human-machine interaction.”Aram Galstyan
Building on IARPA’s previous projects, which explored both automated prediction models and “crowd wisdom,” the new program seeks to blend the strengths of human- and machine-based systems.
In this initiative, the forecasters will use an interactive platform created by the ISI research team to efficiently search and organize information, including traditional news reporting, social media and financial indicators. The platform will also enable the forecasters to interact with machine models, by fine-tuning or adjusting the model-generated forecasts.
Galstyan and ISI computer scientists Emilio Ferrara, Fred Morstatter and Pedro Szekely will focus their research efforts on the study’s technical aspects, including:
- Developing incentive mechanisms to encourage participants to engage in the forecast generation;
- Building interpretable and intuitive predictive models that can be used by human forecasters who do not necessarily have expertise in machine learning; and
- Designing and conducting experiments to study different aspects of human-machine interactions.
Symbiotic interactions
The approach will be validated using randomized control trials; forecasts made with the hybrid systems will be compared with those made by a control group of human-only forecasters. Accomplishing the program’s ambitious goals requires the team to understand the complex and nuanced interplay between humans and machines.
“Fundamentally, this program is about human-machine interaction,” says Galstyan.
“If we want to have symbiotic interactions between the human forecasters and the machines, we need to develop a deep understanding of those interactions. For instance, what are the drivers and determinants of human trust or mistrust in machine-generated forecasts?”
Another key challenge is scalability.
“By the end of the program, our hybrid forecasting system needs to be capable of handling at least 500 individual forecasting problem a year, which is significantly more than was processed in the previous program,” says ISI computer scientist and team member Fred Morstatter.
“If we can combine human judgment with machine-based models in a synergistic way, we can potentially improve the efficiency of the forecasting and tackle a greater number of forecasting problems, while producing better quality forecasts at the same time.”
Published on October 11th, 2017
Last updated on December 5th, 2017
