Nearly 300 times.
That’s how many near aircraft collisions have happened in the United States in the last 12 months, according to the New York Times.
Airline travel has long been considered one of the safest forms of travel. As statistician Arnold Barnett told ABC News: “If you take one flight a day, you would on average need to fly every day for 55,000 years before being involved in a fatal crash.”
But this safety record masks a worrisome trend.
In July, a Southwest Airlines pilot avoided a collision by seconds. Nine days later, multiple jet planes in San Francisco were reported as nearly “skin to skin.” That month alone, there were at least 46 close calls with commercial airlines.
Said Pete Buttigieg, U.S. secretary of transportation, at last March’s Federal Aviation Association (FAA) Summit: “We can’t wait for the next catastrophic event to seek the warning signs of today, fully determine the contributing factors, and swiftly address them.”
Graduate students from the USC Viterbi School of Engineering are doing just that.
Two USC student teams presented their research at the 2023 FAA Data Challenge in Washington D.C.
Vivian Lin, Andrew Bruneel and Tiffany Hoi Ching Wong first heard about the Federal Aviation Association’s (FAA) Data Challenge one year ago, as part of their Applied Data Science master’s program. Lin had been notified by Yolanda Gil, USC Viterbi research professor of computer science, who, in addition to serving as USC Viterbi’s director of new initiatives in AI and data science, had helped draft the nation’s AI roadmap as president of the American Association for Artificial Intelligence (AAAI).
The national competition required students to use real data and produce creative ways in which AI can create solutions for aviation challenges. The top ten 10 university teams would be invited to present their solutions in Washington, DC and compete to win $25,000.
Once mobilized, the students immediately faced their first problem: choosing a subject within the vast field of aviation safety that fit the vague prompt of AI solutions for aviation challenges.
“We had all worked on some sort of aviation projects in the past,” Lin explained, noting that none had a direct experience with aviation safety. “We were supposed to leverage AI and machine learning to address aviation safety issues, but because the prompt was so open ended, and not very specific, we relied a lot on the help of Daniel Scalese [USC program administrator for the USC Aviation Safety and Security Program], as our main point of contact throughout the project.”
The students also had another ally — one who has spent the past 35 years at USC teaching and conducting research on human factors and the safety culture of everything from aviation, the Fukushima nuclear disaster and the Deepwater Horizon oil spill. Professor Najmedin Meshkati, a highly regarded expert in safety culture, has collaborated with major airlines such as Federal Express, Korean Airlines, Turkish Airlines and American Airlines.
It was with the guidance of Gil, Meshkati and Scalese that the students identified runway safety as the main issue they wanted to address with AI.
“There are two main issues in runway safety: incursions and excursions.” Lin explained. “Incursion is whenever two planes on the runway — in take-off or landing — are too close to each other, and it becomes a safety issue. An excursion is whenever a plane lands and runs off its runway.”
Meshkati, in fact, has been conducting research on runway incursions for more than three decades, having been chosen by the Government Accountability Office for a 2007 study on “Aviation Runway and Ramp Safety.”
The student’s plan was to first determine the underlying factors behind runway incursions, runway excursions, and unstable approaches based on the existing data sources, then use artificial intelligence to reveal useful patterns.
The team quickly discovered that the most robust data sets of flight incidents, such as the FAA’s Aviation Safety Information Analysis And Sharing (ASIAS) System, were all confidential and non-accessible to the public. Lin and Wong explained that because the reports contain specific information about the airlines and pilots involved, the data has highly restricted access, making finding other well maintained public datasets a primary issue for the team.
The students eventually found the Aviation Safety Reporting System (ASRS) which is a public, self-reported dataset supported by NASA and contains all incursion and excursion reports from 2000 through 2019. The group quickly discovered, however, that although this public dataset for reporting aviation safety incidents existed, the reporting and collection was often delegated to third party companies, resulting in a lot of the listed categories in the data catalogue not actually existing in the datasets themselves.
“Identifying a good data set was the most challenging,” Lin recounts. “This was our first experience working with real world data. Previously, we had all been doing academic projects, so the data is very clean, it’s controlled and everything is very well labeled. But not in the dataset we were working with.”
The second major challenge the group faced was the content of the data base. Because many of the incidents are self-reported, the data included qualitative descriptions from pilots and air traffic controllers, which required qualitative analysis and the use of natural language processing which the students had to learn for themselves as they completed the project.
In the end, the students used three different flight datasets: qualitative testimonials, more qualitative flight data, as well as weather reports to process and link all the incursion and excursion incidents together to complete their analysis.
The work highlighted several key findings: The students used clustering models to identify keys words across incident testimonials, finding that effective communication, aircraft control and adherence to airport procedures are all key in maintaining safe airport operations and avoiding incursions and excursion incidents. But more significantly, they concluded that the ASRS database would greatly benefit from a vocabulary standardization, and that, in its current state, extensive pre-processing of data is necessary to tease out effective conclusions.
Based on their initial analysis, the group identified a theme of “unstable approaches”— landings in which aircraft do not maintain certain essential flight parameters within reasonable limits, such as the speed or descent rate being too high as they approach the runway — may be a contributing factor to runway incursions and excursions.
Because the project focused on developing machine learning models for evaluating incursion and excursion factors, most of the teams’ findings regarded the quality of the datasets themselves, highlighting necessary policy changes and practices. When evaluating unstable approaches, the students found that the database lacked notation and standardization — explicit documentation on whether altitude values are pressure corrected, and unique identifiers of flights for easier identification — and concluded that the current cutoff of unstable approaches may be too rigid for manual piloting.
The students urged for new standards to be considered for stronger analysis of risks and issues.
This past summer, the students traveled to Washington D.C. to present their work for the final round of judging, all scrambling to balance their internships with the competition that had started nearly nine months before.
The project was very well received, with several judges remarking that the FAA needed more data scientists and thanked them for highlighting important issues in data collection and reporting, even stating that they wanted to do similar work on their own private databases. However, the judges had some concerns with the policy changes the team was suggesting. Stating that in an industry like aviation, especially when incidents are self-reported, the FAA had to be careful where they placed blame.
Said Wong, “In our academic work, we never have to be concerned about the privacy of the sources of the dataset. And after doing this, we found that there are other kinds of issues in reality that are not present in our schoolwork.”
The USC team, according to Meshkati, was the only team that addressed runway safety issues.
Despite not taking home the $25,000 prize, the students still learned essential skills in applied data science. “This project inspired me a lot,” Wong said. “Whenever we start a project, I now know it’s so important to see if the data is of high quality. I was very surprised how messy and how much data was missing within the aviation industry.”
In addition, USC was the only university with more than one team in the FAA Data Challenge 2023 finals.
A second USC Viterbi team also participated in the data challenge, their research focusing on the increasingly pressing issue of travel flight delays and scheduling issues.
Post-pandemic, it seems like every holiday weekend is now met with a series of headlines warning of canceled and delayed flights, hours of TSA lines, and many unhappy travelers not making it home for the holidays.
A multidisciplinary team of USC Viterbi students, including undergraduates Reagan Arvidson (astronautical engineering) and Nicholas Lototsky (aerospace engineering); Kelli McCoy, a Ph.D, candidate in environmental engineering; and Emma Silverstein, a master’s student in civil engineering, developed a machine learning model to address the ongoing issues with flight scheduling. They also investigated modifications to prior flight schedules, addressing issues of airspace in a manner that adhered to safety and emission requirements.
The team was formed and advised by Roger Ghanem, professor of civil and environmental engineering and aerospace and mechanical engineering, who reached out to Silverstein, a student in his Risk Analysis in Civil Engineering class.
Silverstein reported that her team similarly struggled with utilizing real world data. To address the issue of flight paths, airspace and rescheduling, they ended up using NASA’s Sherlock database to sample flight paths and the Aviation Environmental Design Tool (AEDT) to calculate the emissions of airplanes over given flight paths for the varying expected weather conditions.
“We were working with huge data sources, and performing any operations could take a really long time,” she said. But the results yielded a workflow for the assessment of safety risk, whileminimizing emissions and enabling efficient replanning of flight paths in the face of reschedulingneeds like weather delays.
“Our big finding (from the competition) was that our workflow appears to be a viable way to reroute flights.” Silverstein says, “It might not be practical yet, as it’s still in its early stages.” But she and her teammates are continuing to process more data and learn more about the theories behind their work.
Published on November 17th, 2023
Last updated on November 17th, 2023
