To evaluate the performance improvement of FLASH 2.0, we established a test bed to make apple-to-apple comparisons between SLAM and FLASH on real world TMIs, which are managed in the operation using SLAM.
Every time a TMI is issued, we will take a complete snapshot of the operational data, which is further used to calculate the operational impact of the slot assignment given by SLAM.
We then use the same data snapshot to run FLASH 2.0 to compute the operational impact of the slot assignment given by FLASH.
So for each SLAM run, we can make such comparison. Thanks to the Transform generation functionality on Foundry, we are able to create the data transforms dynamically in batches. Basically we can specify the input/output files in a config file and iterate though it can create data transforms for each SLAM run file.
With this ability, our test bed is able to test at scale. We tested hundreds of real world SLAM runs among over 50 TMIs, But we don’t wanna the percentage changes shown here get skewed by the extreme values of some small TMIs. So they are only calculated over the 20 major TMIs, happening this year at our hub with at least 150 TMI flights we have observed consistent improvement across multiple aspects. The figure on the right shows average percentage improvements of 8 metrics categorized into the three groups, flight delays, legalities and curfews, and customer.
---As you can see, the improvements are significant. Note that since it is the percentage improvements, the averages are only based on TMIs in this year Because for smaller TMIs, ---the absolute values of some metrics like crew legalities can be small, which make--- the percentage changes might have extreme values like 100% or 0 percentage. We don’t want the average percentages get skewed due to these extreme values. So they are calculated shown here.
As the major objective of FLASH 2.0, the delay metrics on average achieves 20% to 30 percentage reduction,
--- which seems too good to be true. But it is mainly because our benchmark, the slot assignment given by SLAM, did not consider the downline flights at all. So it still kind of makes sense.
For crew legalities, FLASH is able to reduce pilot timeouts by 42% and flight attendant timeout by 35%. And For cut missed airport curfew by 35%.
For customer related metrics, FLASH reduced customer misconnection by 35% and customer delay minutes by 15%. Though 15% is not as significant as the other metrics, but we are talking about customer delay minutes that could be as high as the scale of million. So the improvement in customer delay is also significant.
So this slide details the percentage reduction of missed connection flights for each of the major TMIs mentioned above, averaging at 35% percentage. One thing to note is that the model even has a logic to penalized customer miss conn to low -frequency market more than high-frequency market. Because for low frequency markets like international flights, it would be much more difficult for customers to rebook if they missed the original flight, and their travel experience would be greatly impact if miss connections happen
And this slides details the absolute changes of pilot and flight attendant legality issues. For some TMIs that FLASH 2.0 is able to reduce more than 50 crew legality issues. For others, though the absolute values might be low, but as we showed before, the average percentage reduction is around 35% to 40%, which is definitely significantly. But as we mentioned before, FLASH 2.0 is a multi-objective optimization, so we can reduce the infeasibilities, but cannot complete eliminate them.
Another benefits brought by FLASH is that it can reduce the infeasible slot assignment when an aircraft routing touches the same GDP twice. Because SLAM does not consider the downline flights, so the later leg that is influenced by the GDP might be assigned to a slot that is infeasible given the slot assignment of the earlier leg of the same aircraft. Since FLASH consider the downline flights, it can reduce such infeasible assignments, so our NOC operator could avoid manually adjusting the slot assignments given by FLASH.