FOAMEd review of intubation checklist research

Hi folks

You might have heard Tim Leeuwenberg and I had a debate regarding checklists and airway experts

His side of the debate is well summarised here on his excellent blog

I must say he debated very well and was the clear audience favourite. When Cliff Reid polled the expert panel on their opinion, here were the results.

Did they use airway checklists?
Dr Richard Levitan = NO
Dr Keith Greenland = NO
Dr Scott Weingart = YES

Anyway I was surprised since the debate that an Article in Press in the American Journal of EM came out on perintubation checklists!
Here it is

Impact of checklists on peri-intubation care in ED trauma patients

Key points about the study:
– retrospective, before and after analysis
-underpowered to show outcome differences
– RSI rates significantly more post checklist
– no significant difference in intubation attempts nor success
-no significant difference in haemdynamics nor oxygen desaturation rates with use of checklist
– Mix of anaesthesia and EM docs prior to CL , then 99% EM docs doing intubations post CL so potential confounder

I sent out a call to FOAMEd airway folks to review the is what they said!

Hi Minh

First glance:
Not clear when CL was used – either pre, during or post RSI
“CL” may have positive influence on post-intubation sedation, but underpowered to see if effects on LOS/mortality etc
Looks more like an audit of activity in a unit than a proper challenge-response checklist in a tightly-coupled procedure by trained personnel
Will cogitate and get back to you – unless want to share comments via G+?
Stil think the question of CL use is best answered by experts in human factors, not airway…perhaps ask James French too?
More importantly, is your wife allowing you to engage in another CL debate after the last fracas?
Cheers mate
Tim Leeuwenburg
Kangaroo Island
Twitter @KangarooBeach
And now from Dr Yen Chow in Canada!
I agree with Tim. 

I would argue that their intervention does not appear to be designed as a checklist to be used at the time of intubation, whether pre-intubation or immediately post intubation. The form appears to be designed as a quality audit form. The checklist items appear to be optional and are not challenge-response. The authors also agree that they cannot tell if the providers only used a portion or all of the form.  The “checklist” does not appear to engage the team in what the approach might be nor plan for failure. Given these design issues, it is highly suspect that this was used as an intubation checklist which in most practitioners’ minds would occur before intubation. It also leads me to wonder what was the education on the use of the “checklist” as well?

The effect of more RSI and more postintubation sedation observed after introduction of this audit might be attributed to an observer effect of the study itself.
Dr Yen Chow


Then this from Dr Peter Fritz, Melbourne

Hi Guys, 

Not much of airway/intubation checklist IMHO.
Lots of stuff missing
  • no mention of team
  • no mention of plan A
  • no mention of plan B/C/D etc
  • no mention of patient, equipment, optimisations etc
Not sure if this paper adds much to what we know about checklists in critical care.
Thinking about my favourite quote from The Checklist Manifesto: How to get things right by Gawande is:
“The checklist gets the dumb stuff out of the way, the routines your brain shouldn’t have to occupy itself with and lets it rise above to focus on the hard stuff ( i.e. Where should we land?)”
This checklist doesn’t do that.
Then the usual essay from Dr Nicholas Chrimes, Melbourne
Agree with all the previous comments. Overall this is a flawed “checklist” and a flawed study. A few specific points:

  • There is a major change in practice coincident with the introduction of the checklist with 1/3 of intubations being performed by anesthesiologists pre-checklist compared with almost 100% by EP’s post checklist. This kind of confounder would seem to make the impact of the checklist on the outcomes listed uninterpretable. Subgroup analysis of the EM vs anaesth data might have sorted this out but other than for the “increased RSI” outcome, this info has not been provided. Thus it may be that there were major changes in outcomes from the checklist within these groups that were negated overall by the concomitant change in the nature of the airway provider. This issue isn’t mentioned at all in the “limitations of the study” section.
  • There’s no definition of what a checklist is (I agree with Tim, what they’ve provided looks more like an audit tool), how it was used or how staff were trained to use it. The structure of the CL, timing of implementation and the nature of the training provided is crucial to the ability of a CL to have an impact. I’m not convinced they’ve properly implemented it or that it even constitutes what we would call a checklist.
  • Not only do we not know when the “checklist” was used the “intention to treat” format means that there’s no data provided that tells us if the checklist was EVER used. The discussion seems to imply that it was used but how often, when and to what extent is unclear. The reason many metrics showed no change might reflect the fact that the checklist was never used – or was used after the fact. I understand this is the point of “intention to treat” but it would have been nice to get some sense of whether the failure to produce a change in some of their outcome measures was due to failure to use the checklist.
  • I’m not sure their outcome measures are the right ones. Many of the metrics they’re looking don’t seem to be the sorts of things we are intending to address with checklists (eg. BP/HR changes, length of ED stay, days on a ventilator, etc). Given we all know that despite doing everything poorly it is still possible to have a reasonable outcome, I think assessment of CL’s requires more outcomes that assess process rather than outcome. For example if cricoid was intended but forgotten, the suction not connected or the spare laryngoscope not working – most of the time these things won’t matter anyway (especially the cricoid I hear you all say!). Assessment of checklists need to measure preparedness to deal with rare events, not just the outcomes – as without this if the rare event doesn’t occur with sufficient frequency during the study, no impact will be detected anyway). 
  • It’s also not clear what they’re defining as an “RSI” (PreOx? Use of sux? Use of cricoid? Mode of administration of drugs?) and thus whether this is significant. The fact that their RSI rate was only 75% pre-checklist seems a bit odd to me if the alternative was something other than a “modified RSI” and would seem to indicate an underlying education problem.  Again recognising that a patient needs RSI and remembering to implement it is not typically what you’d envisage a checklist being used for. It’s hard to see how this (their only positive finding) could be attributed to the checklist provided. 
To be blunt, I have to say that this study adds nothing to our understanding of the ability of checklists to impact on patient care.
Then Dr Richard Levitan, who needs no introduction.
well said by all; checklists “sexy” –but getting them right, useful, short, and actionable…aye, there’s the rub!

and they’re just one part of over engineering controls–i.e. make tools easily accessible, have things that work passively, deploy back-up/safety things in tandem, etc…..which just got me thinking—another beautiful thing about NO DESAT—it’s already in place before the missed attempt….i.e. its deployed in moment of calm, without operator having to remember it in crisis


Richard M. Levitan, MD
Airway Cam Tech., Inc.;
610-639-7706 (cell)
610-341-9560 (voice); 610-341-1866 (fax)
PO BOX 337 Wayne PA 19087

Practical Emergency Airway Course:

20 cadavers – Incredible imaging
Real tissue & Practical Approach
Yellowstone and NYC Advanced Airway Courses:
Advanced AIrway Techniques & Critical Care
Jackson Lake Lodge (WY) and Millennium Hilton (NYC)
And finally from Dr John Vassiliadis, Sydney
Dear all,

Happy Easter,
sorry for the delay in replying.. was busy with family things…
I agree with the comments already made, I think this was simply an audit of what they did and what effect having EM physicians have made to their department.
I think from what they have written you can not differentiate EM influence from their very basic checklist….
No discussion of how they came up with the checklist and how they agreed upon it  and how it was enforced.
Very small study.. not convinced it add anything to the debate.


2 thoughts on “FOAMEd review of intubation checklist research

  1. Must….resist…..engaging in ‘debate’

    This study is u derpowered and more about ise of an audit tool than of a proper checklist

    The experts in checklist use are human factors bods (thanks to Nick Chrimes for that)

    Lets get that nice James French on and podcast about it if you REALLY want to talk about checklists and crisis management in EM-Anaes….

    I promise no boiling of urine…but is corngrillin’ allowable?

    Later. Burg not berg….

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: