Thank you for your hard work. We completed our manual evaluation for 274 ATC/ATC-INFO messages on v2.6.0 (branch GENAICOE-672). Feedback items from 1/8-1/9 flights (examples are provided in the Excel file):
- GenAI misses some straightforward ATC location/reason information in Flight Feed Examples are provided in the attached Excel file. Please try to tweak the prompts so GenAI can pick out these information more consistently.
- When GenAI makes up ATC location/reason information even thought there is no such information in Flight Feed: It seems that the raven keywords of such examples are all “INFO ATC GS Ground Stop”; and most of the hallucination phases are “in Houston” and “due to weather”
- GenAI picks up information in Flight Feed which does not make sense as ATC reason or articulates the reason in a weird way Examples are provided in the attached Excel file. We suggest only include the reason in the message when the ATC programs are due to weather.