Latest (2012) Nall Report

[rwtucker]

I just downloaded a copy of the 24th Joseph T. Nall report on aviation safety, published by the AOPA Air Safety Foundation (there is no 2011 report). The report is too large to upload here but I got mine from this link. http://www.aopa.org/-/media/Files/AOPA/ ... all-V6.pdf PM me if that link doesn't work -- it might be membership linked.

I have had only enough time to give the report a quick scan. Even so, this year's report appears to carry forward the same emphasis on cross-tabs by categories that are not appropriately corrected for proportionality. The signal that this would continue to be the case can be found in the Publisher's Forward (page 7) where we are presented with the following behaviorally useless information about leading causes of fatal accidents.

Figure 1: US Fatalities in 2012 Accident Type Deaths
- Driving 33,561
- In-Home Accidents 18,000
- Motorcycling 4,957
- Swimming 3,533
- Cycling 726
- Boating 651
- General Aviation 378
- Lightning Strikes 28

There is some talk about accident rates (per 100,000 hours) but this framework is only partially and inconsistently explored in the text while the tables compare raw accident rates for very unequal population distributions. For example, Table 15 (page 21) shows a potentially interesting category of total and fatal accidents by certificate level. A logical next step would be to look for a Table 16 showing accidents by certificate level adjusted for hours flown by type, etc. Don't look. Table 16 goes on to a different topic. The accident by qualifications issue is explored in the text in an incomplete and confusing way as follows:

PILOT QUALIFICATIONS Nearly half of all
accident flights were commanded by private pilots
(Figure 15), including 52% of fatal accidents. Thirty
percent were flown by commercial pilots and 15%
by ATPs. Fifty-eight percent of all accident pilots
were instrument-rated, slightly less than the 64%
of all pilots with private or higher certificates who
held that rating in 2012. However, that population
includes commercial and airline transport pilots
who do little or no GA flying beyond positioning legs
flown under Part 91 in company aircraft. Restricting
the comparison to private pilots shows similarly
small differences but in the opposite direction: One
third of the accident pilots were instrument-rated
compared to 28% of private pilots nationwide.
For the second consecutive year, higher certificate
levels were associated with reduced lethality. This
stands in contrast to previous years, when there
was little apparent difference between certificate
levels. Only five of the 77 accidents on student solos
were fatal.

Don't look for any of the accident by manufacturer/aircraft type data we saw in some older reports. It's not there. It may come out in subsequent reports.

I'm going to give it a good read tonight.

[Colin]

After reading The Killing Zone the stat that my brother and I really wanted to see was "hours in the last 90 days in type." It looks like it is often really low.

[Gnomad]

After reading The Killing Zone the stat that my brother and I really wanted to see was "hours in the last 90 days in type." It looks like it is often really low.

Read that book very early on after obtaining my PP. Was flying my DA40 multiple times per week, so the 90 day thing didn't cross my mind, but surpassing 350 hours was a big milestone moment for me. Statistically, on that day, it was a sign that I kind of knew what I was doing!

[rwtucker]

Colin & Eric,

I know what you guys are saying! Even though I crossed 1,000 hours TT a while ago, i have been very low time the past six months. I got in the DA40 a few weeks ago for a routine flight, everything felt different and a bit unsafe. I remember my CFI reminding me, Its not like riding a bicycle.

[Colin]

Every time I get in the plane, I pause with my foot on the step and my hand on the glare shield and I think these four words: "Is today the day?"

I keep expecting the failure. It is not the most relaxing way to fly, but both times I have had an *actual* failure have been non-events. (Both were loss of electrical power, I have yet to join the dead stick club.) The pilot friend who was with me on final into KLGB when the screens and radios went dark said, "You didn't bat an eye, you didn't shift in your seat or *anything,* you just kept flying."

That's what they taught me. And I follow the rule that if I haven't flown in a week I can't fly the family. That keeps me in the sky.

[CFIDave]

As a pilot only two bad things can happen to you and one of them will be:

a. One day you will walk out to the aircraft knowing that it is your last flight.

b. One day you will walk out to the airplane not knowing that it is your last flight.

[Colin]

I'll never understand the Wright Brothers, because they did it before anyone, mastered it, and then neither ever flew again. So strange.

My family medical history does not suggest I will get to do it until I die, but I would like to do it until my sons or grandchildren take over and start flying me around instead.

[Lance Murray]

For those of you mathematically inclined (statistics) here is a great response to The Killing Zone.

http://www.sciencedirect.com/science/ar ... 7513003242

Highlights
•
A serial-nonlinear model appears to predict general aviation accident rates from total pilot flight hours despite the inherently noisy nature of such data.
•
This could serve as an improved independent variable or covariate to control for flight risk during data analysis of aviation accidents.
•
Applied to FAA data, this class of models implies that GA pilots may face elevated flight risk longer than imagined before leveling off to a baseline rate.

[rwtucker]

It is a longer discussion . . . but this is neither a particularly robust nor logically exhaustive analytic model. The model and the logic are separate points. Properly included, the "noise" becomes part of the larger error term associated to analyzing discrete categorical data of this kind and increases variance. Too many analysts looking to prove a point exclude non-conforming cases as outliers until the plots begin to look like what they want. From the abstract, I cannot see how this was dealt with nor can I see the rationale for the weightings. These contexts represent opportunities for researcher bias to creep in. I note that the confidence band seems unusually narrow for these kinds of data and presumably reflect corrections that are probably specified in the full article. In addition, the R^2 (coefficient of determination) leaves a pretty large FVU and raises a question as to whether we are seeing the output of an "IV hunt." If so, there is a larger context in which the threshold for significance should be adjusted for the number of IVs examined that failed to show significance. These IVs are often excluded when findings are reported. BTW: The difference in the initial slopes of the curves between IFR and non-IFR might reflect or partially reflect an unaccounted for lag function representing the difference in training and supervised time.

All of this is easily misinterpreted in these short comments. I'll summarize by saying that this study means a lot less than some might think, especially in practical terms.

My take:

The generalization from this analysis is that there is a range of hours during which your theoretical risk of an accident rises somewhat sharply (albeit by a very small real amount), then declines more slowly from a fairly narrow peak. Similarly, there are ranges of hours during which your theoretical risk of an accident is lower and, post-peak, somewhat more flat over hours of flight. The post-peak range is open-ended because the analysis that produced the Goodness of Fit estimates stops at 2,000 hours. Since the accident rate is non-zero at 2,000 hours, it is possible that the data were cut off because the model broke beyond 2,000 hours (e.g., R^2 dropped precipitously); and/or, there may not have been enough cases to produce a good analysis.

Taking these generalizations as provisionally sound, my question is, "So what?" Am I going to be more careful during my theoretical period of higher risk (which the authors acknowledge may not fit me because of the large error term)? If so, this implies that I am going to be less careful at other times! Hmmmm.

There are some other interesting findings here. Note the very small and perhaps practically meaningless difference between the peak accident rate and what you can visually infer as the aggregate mean rate. Does this very small rate difference justify a behavioral change? Can you see it contributing to a model applicable to individual pilots? Separately, note the large portion of the curves for which non-IFR pilots appear to fly safer than IFR rated pilots. We need to learn more about what we see in the internals of these curves. Also notice that the scaling interval for the two curves has been manipulated to make the curves look identical (this is frowned upon in scientific reporting standards).

[Lance Murray]

Here is the FAA report.

https://www.faa.gov/data_research/resea ... 201503.pdf