Risky Business: Framing Statistics

The Gist: Medical literature frequently frames effect size using relative and absolute risks in ways that intentionally alter the appearance of an intervention, an example of framing bias. Effect sizes seem larger using relative risk and smaller using absolute risk [1,2].

Part of our 15 Minutes - 'Stats are the Dark Force?' residency lecture series
Risk from Lauren Westafer on Vimeo.

Perneger and colleagues surveyed physicians and patients about the efficacy of 4 new drugs. They were presented with the following scenarios:
While these numbers reflect the same data, patients and physicians selected that the drug that "reduced mortality by 1/3" was "clearly better" [3].  This indicates both parties are susceptible to the cognitive tricks statistics, particularly the way that relative risk make numbers appear larger and the ways absolute risk makes numbers appear smaller.  Authors can consistently switch between relative and absolute risk to maximize the appearance of benefit and minimize risks, as seen in the following abstract:

Perhap authors should be encouraged to report statistics consistently, using relative OR absolute rather than switching between the two to give the appearance of maximum benefit and minimal risk.

References:
1. Barratt A. Tips for learners of evidence-based medicine: 1. Relative risk reduction, absolute risk reduction and number needed to treat. Canadian Medical Association Journal. 171(4):353-358. 2004
2. Malenka DJ, Baron JA, Johansen S, Wahrenberger JW, Ross JM. The framing effect of relative and absolute risk. Journal of general internal medicine. 1993;8(10):543-8.
3. Perneger TV, Agoritsas T. Doctors and patients' susceptibility to framing bias: a randomized trial. Journal of general internal medicine. 26(12):1411-7. 2011.

P Values: Everything You Know Is Wrong

The Gist:  P values are probably the most “understood” statistic amongst clinicians yet are widely misunderstood.  P values should not be used alone to accept or reject something as “truth” but they may be thought of as representing the strength of evidence against the null hypothesis [1].

Part of our 15 Minutes - 'Stats are the Dark Force?' residency lecture series

P: Everything You Know Is Wrong from Lauren Westafer on Vimeo.


At various times in my life I, like many others, have believed the p value to represent one of the following (none of which are true):
    • Significance
      • Problem:  Significance is a loaded term.  A value of 0.05 has become synonymous with “statistical significance.”  Yet, this value is not magical and was chosen predominantly for convenience [3].  Further, the term “significant” may be confused with clinical importance, something a statistic cannot answer.
    • The probability that the null hypothesis is true.
      • Problem: The calculation of the p value includes the assumption that the null hypothesis is true.  Thus, a calculation that assumes the null hypothesis is true cannot, in fact, tell you that the null hypothesis is false.
    • The probability of getting a Type I Error 
      • Background: Type I Error is the incorrect rejection of a true null hypothesis (i.e. a false positive) and the probability of getting a Type I Error is represented by alpha. Alpha is often set at 0.05 so that there is a 5% chance you are wrong if you reject the null hypothesis. This is a PRE test calculation (set before the experiment)
      • Problem: Again, the calculation of the p value assumes the null hypothesis is true. The p value only tells us the probability of getting the data we did, it does NOT speak to the underlying truth of whatever is being tested (i.e. efficacy). The p value is also a POST test calculation.
      • The error rate associated with various p values varies, depending on the assumptions in the calculations, particularly prevalence. However, it's interesting to look at some of the estimates of false positive error rates often associated with various p values:  p=0.05 - false positive error rate of 23-50%; p=0.01 - false positive error rate of 7-15% [5].
P value is the probability of getting results as extreme or more extreme, assuming the null hypothesis is true. Originally, this statistic was intended to serve as a gauge for researchers to decided whether or not a study was worth investigating further [3].  
  • High P value - data are likely with a true null hypothesis [Weak evidence against the null hypothesis]
  • Low P value - data are UNlikely with a true null hypothesis [Stronger evidence against the null hypothesis]
Example:  A group is interested in evaluating needle decompression of tension pneumothorax and proposes the following:
    • Hypothesis - Longer angiocatheters are more effective than shorter catheters in decompression of tension pneumothorax.
    • Null hypothesis - There is no difference in effective decompression of tension pneumothorax using longer or shorter angiocatheters.
A group, Aho and colleagues, did this study and found a p value of 0.01 with 8 cm catheters compared with 5 cm catheters.  How do we interpret this p value?  
  • We would expect the same number of effective decompressions or more in 1% of cases due to random sampling error.  
  • The data are UNLIKELY with a true null hypothesis and this is decent strength evidence against the null hypothesis.
Limitations of the Letter “P”
    • Reliability.  P values depend on the statistical power of a study. A small study with little statistical power may have a p value greater than 0.05 and a large study may reveal that a trivial effect has statistical significance [2,4].  Thus, even if we are testing the same question, the p value may be "significant" or "nonsignificant" depending on the sample size.
    • P-hacking.  Definition:  "Exploiting –perhaps unconsciously - researcher degrees of freedom until p<.05" Alternatively: "Manipulation of statistics such that the desired outcome assumes "statistical significance", usually for the benefit of the study's sponsors" [7].
      • A recent study of abstracts between 1990-2015 showed 96% contained at least 1 p value < 0.05.  Are we that wildly successful in research? Or, are statistically nonsignificant results published less frequently (probably).  Or, do we try to find something in the data to report as significant, i.e. p-hack (likely).
P values are neither good nor bad. They serve a role that we have distorted and, according to the American Statistical Association: The widespread use of “statistical significance” (generally interpreted as “p ≤ 0.05”) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process [1].  In sum, acknowledge what the p value is and is not and, by all means, do not p alone.

References:
  1. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat. 2016;1305(April):00–00. doi:10.1080/00031305.2016.1154108.
  2. Goodman S. (2008) A dirty dozen: twelve p-value misconceptions. Seminars in hematology, 45(3), 135-40. PMID: 18582619 
  3. Fisher RA. Statistical Methods for Research Workers. Edinburgh, United Kingdom: Oliver&Boyd; 1925.
  4. Sainani KL. Putting P values in perspective. PM R. 2009;1(9):873–7. doi:10.1016/j.pmrj.2009.07.003.
  5. Sellke T, Bayarri MJ, Berger JO.  Calibration of p Values for Testing Precise Null Hypotheses.  The American Statistician, February 2001, Vol. 55, No. 1
  6. Chavalarias D, Wallach JD, Li AHT, Ioannidis JPA. Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. Jama. 2016;315(11):1141. doi:10.1001/jama.2016.1952.
  7. PProf.  "P-Hacking."  Urban Dictionary. Accessed May 1, 2016.  Available at: http://www.urbandictionary.com/define.php?term=p-hacking

Percutaneous Chest Tubes: The Humane Choice

The Gist:  Small bore percutaneous catheters, often referred to as "pigtail" catheters, should be the initial means of treating many pneumothoraces and select other drainable thoracic pathologies as they cause less pain and capitalize on the commonly used seldinger technique [1-10].

Traditional tube thoracostomy is an invasive procedure.  For the past several years, international guidelines, individuals in the Free Open Access Medical education (FOAM) community, and various institutions have moved towards placing more pigtail catheters for urgent thoracic pathology.  Yet, this practice is still not ubiquitous.  I recently gave a talk on this to my program and, in the spirit of FOAM, have shared it:

Technique - Watch this video by Dr. Larry Mellick 
  • Seldinger style: uses a technique with which we are intimately familiar. The majority of emergency providers have likely done far more central lines than open tube thoracostomies. As such, a technique mentally and mechanically familiar to providers may be preferable.
  • Pearls for placement - Kulvatunyou and colleagues suggest "POW" pearls for placement.
    • P -Perpendicular: Ensure the finder needle is perpendicular to the rib during placement
    • O -Over the rib: Like chest tubes, pigtails go over the rib to avoid injury to the neurovascular bundle
    • W -Wary of wire kinking:  The wire may be prone to kinking, particularly upon dilation through the tough intercostal muscles.
Indications
  • Pneumothorax
    • Spontaneous pneumothorax: The British Thoracic Society has recommended small bore tubes over traditional chest tubes since 2010.
    • Traumatic pneumothorax:  Use of pigtail catheters have increased in many trauma communities, with success rates comparable to large bore chest tubes [8-11].
  • Effusions - pigtail catheters are frequently used to drain effusions, particularly simple effusions. Most of the primary literature on this topic has been conducted in children with parapneumonic effusions and has demonstrated that this technique is successful and safe [13].
Cautions/Contraindications
  • Hemothorax/Complex fluid - Larger bore tubes (28F and larger) are typically used to drain hemothorax due to the feared complication of retained hemothorax.  A prospective review of 36, 14F pigtail catheters placed for hemothorax in trauma patients found no significant differences in complications or success between pigtails or chest tubes but wasn't powered to find important, infrequent complications [11].  An animal study found
The Good:
  • Less Painful - In addition to the procedure not requiring large, forceful separation of the and unsurprisingly, placing a smaller tube in the chest causes less pain, even 2 days after the procedure [10].
    • Pain in pigtail vs chest tube patients: Day 0 3.2  vs 7.7; p<0.001, Day 1 1.9 vs 6.2; p<0.001, Day 2 2.1 vs 5.5; p=0.04 (note: no power calculation performed)
  • Easy/Familiar Procedure - as above under "Seldinger technique"
  • May reduces some complications - The literature suggests that complications are typically at least equivalent between larger chest tubes and pigtails. More serious complications are difficult to quantify given the infrequency.
    • One study did show that infections were reduced in the pigtail group, possibly due to technique or a larger nidus for infection [2].
  • Outpatient treatment possible - In select patient groups with spontaneous pneumothorax and excellent follow up, a pigtail catheter may be connected to a heimlich valve and the patient may be discharged [7].
The Bad:
  • More predisposed to kinking - Due to the small, flexible tubing, these tubes may kink and obstruct the lumen.  The trauma literature suggests these complications may occur in 2-8% of cases [8-10].
  • Clogging - Drainage of some complex fluids (loculated effusion/hemothorax) may be more problematic through pigtail catheters as the small lumen may be easier obstructed with clot.
  • Time? The belief exists that open thoracostomy more expediently relieves pneumothorax compared with the percutaneous technique and is thus preferred in emergent, life-threatening situations. To date, there's no literature to support or refute this and the time a tube takes is likely provider dependent.
  • It's less cool - A certain pride and thrill exists with performing invasive procedures.  In discussions with individuals regarding barriers to uptake of the percutaneous technique the theme arose that performing this technique would demonstrate some sort of weakness by the provider. Note: this notion is not supported or addressed by the literature and is merely a thought about subconscious provider bias
References:
1. Laws D et al. BTS guidelines for the insertion of a chest drain. Thorax. 2003 May;58 Suppl 2:ii53-9.
2.  Benton IJ, Benfield GF. Comparison of a large and small-calibre tube drain for managing spontaneous pneumothoraces. Respir Med. 2009 Oct;103(10):1436-40.
3. Dull KE, Fleisher GR. Pigtail catheters versus large-bore chest tubes for pneumothoraces in children treated in the emergency department. Pediatr Emerg Care. 2002 Aug;18(4):265-7.
4. Gammie JS et al. The pigtail catheters for pleural drainages: a less invasive alternative to tube thoracostomy. JSLS. 1999 Jan-Mar;3(1):57–61.
5. Kuo HC, et al. Small-bore pigtail catheters for the treatment of primary spontaneous pneumothorax in young adolescents. Emerg Med J. 2013 Mar;30(3):e17.
6.  Repanshek ZD, Ufberg JW, Vilke GM, Chan TC, Harrigan RA. Alternative Treatments of Pneumothorax. J Emerg Med. 2013 Feb;44(2):457-466.
7. Hassani B, Foote J, Borgundvaag B. Outpatient management of primary spontaneous pneumothorax in the emergency department of a community hospital using a small-bore catheter and a Heimlich valve. Acad Emerg Med. 2009 Jun;16(6):513-8.
8. Kulvatunyou N, Vijayasekaran A, Hansen A, et al. Two-year experience of using pigtail catheters to treat traumatic pneumothorax: a changing trend. J Trauma. 2011 Nov;71(5):1104-7.
9. Rivera L, O’Reilly EB, Sise MJ, et al. Small catheter tube thoracostomy: effective in managing chest trauma in stable patients. J Trauma. 2009 Feb;66(2):393–9
10.  Kulvatunyou N, et al. A prospective randomized study of 14-French pigtail catheters vs 28F chest tubes in patients with traumatic pneumothorax: impact on tube-site pain and failure rate. EAST Annual Surgical Assembly, Oral paper 12, Jan 17, 2013.
11. Kulvatunyou N, Joseph B, Friese RS, et al. 14 French pigtail catheters placed by surgeons to drain blood on trauma patients. J Trauma Acute Care Surg. 2012;73(6):1423–1427. 
12. Russo RM, Zakaluzny SA, Neff LP, et al. A pilot study of chest tube versus pigtail catheter drainage of acute hemothorax in swine. J Trauma Acute Care Surg. 2015;79(6):1038–1043. 
13.  Liu YH, et al. Ultrasound-guided pigtail catheters for drainage of various pleural diseases. Am J Emerg Med. 2010 Oct;28(8):915-21
14. Inaba K, Lustenberger T, Recinos G. Does size matter? A prospective analysis of 28-32 versus 36-40 French chest tube size in trauma. The journal of trauma and acute care surgery. 72(2):422-7. 2012.

We Don’t Know the Midclavicular Line

The Gist:  Needle decompression for tension pneumothorax should be taught at the fourth or fifth intercostal space at the anterior axillary line (4/5ICS AAL). 

  • Note: This post will not detail critiques that needle decompression may be overused or the needle vs thoracostomy debate.

Historical teaching instructs providers to place a needle in the second ICS at the mid-clavicular line (2ICS MCL) for tension pneumothorax [1,2]. Free Open Access Medical Education (FOAM) sources such as Emergency Medicine Ireland have preached the more lateral approach for years; yet this teaching has not spread widely (outside of military circles where there seems to be better adoption). Change is difficult, particularly when it involves re-educating thousands of providers and it seems like this is the primary driver behind the 2ICS MCL remaining as the typical site for needle decompression..  However, several potential problems exist with the mid-clavicular approach that warrant consideration for assuming 4/5ICS AAL as the primary initial placement for needle decompression.

A: Where I see most needles placed, B: 2ICS MCL, C: 5ICS AAL
We may not be able to reach the pleura [3-5].  The chest wall may be particularly thick at the 2ICS MCL, particularly as the average BMI in many nations grows.  Researchers have looked at this question for years through a couple of means - measuring the depth at the 2ICS MCL on CT scans of trauma patients compared with alternative sites. The 2ICS MCL is generally 1.3 cm thicker than 5ICS AAL. 
  • This discrepancy was not solely seen in the morbidly obese.  In fact, it was seen consistently across all four BMI quartiles tested, and at the traditional insertion site, needle decompression would have been extremely difficult with any eccentric placement using a standard needle in all but the lowest BMI quartile [3].
How often would the needle fail?  A systematic review and meta-analysis in Injury 2015 by Laan et al looked at 17 studies, generally cadaveric or radiographic, and found that a standard 5 cm catheter used for needle decompression at the 2ICS MCL would fail 38% (24–54%) of the time compared with only 13% (8–22%) at ICS4/5-AAL (p= .01) [5].
  • The British Thoracic Society Guidelines (2010) even remark “a standard 14 gauge (4.5 cm) cannula may not be long enough to penetrate the parietal pleura..with up to one-third of patients having a chest wall thickness >5 cm in the second interspace.”
  • In some places, the failure rate may be even higher secondary to obesity [5].
What about a longer needle?  Many catheters used for needle decompression are 5 cm in length; however, some have access to 8 cm angiocatheters.  A analysis by Clemency and colleagues found that in order to achieve a success rate of 95%, we would need a catheter at lease 6.4 cm in length [8].  Similarly, Laan and colleagues conducted a pre-post retrospective study in an EMS system that switched from using 5 cm catheters to 8 cm catheters with an increase in success rate (48% vs 83%) [6].  For a life saving, last ditch effort, I'm not sure that 95% success rate is adequate when alternatives exist.

We don’t identify this site well [10,11].  A 2005 paper by Ferrie and colleagues had 25 emergency physicians name the correct side for needle thoracentesis and label this site with a pen on a male volunteer (erased between providers).  Nearly all participants were ATLS certified within the past 10 years.  
  • 88% (n=22) named the correct site (one additional person did name the 5ICS AAL).
  • Only 15 of the 25 participants (60%) could correctly identify the 2ICS MCL [10].   
In another study, Inaba and colleagues trained 25 US Navy corpsmen on needle decompression, using both the 2ICS MCL and the 5ICS AAL. The corpsmen then performed needle decompression at both sites on randomly selected cadavers, bilaterally.  
  • Mean distance from the correct location: 3.1 cm 2ICS MCL vs 1.2 cm 5ICS AAL
  • Correct placement (ICS +/- 5 cm):  15/50 (30%) 2ICS MCL vs 41/50 (82%) 5ICS AAL
  • Limitations: This study had multiple outcomes and no power analysis was performed [11]
I think much of this is because we underestimate the length of the clavicle.  It's easier when you can see the chest wall bones but we don't have this advantage in the clinical setting.  On a person, the midclavicular line often seems fairly lateral.  


Important structures surround the 2ICS MCL.  As mentioned above, we seem to have a tough time finding the 2ICS MCL [8,9]. There are important structures in this vicinity, particularly if the tendency is to go more medial than the actual midclavicular line, including the internal mammary artery and contents of the superior mediastinum.  Naturally, should an individual placing a needle in the 4/5ICS AAL go too caudal the possibility exists for the needle to enter the liver or spleen but the study by Inaba and colleagues suggest we may be better able to identify this space [9].

Given the literature, it seems that at this time should a needle be placed aiming for the 2ICS MCL for needle decompression and fail, this is a failure of education and changing our knowledge base rather than a patient-based failure. We should know better.

References:

  1. MacDuff A, Arnold A, Harvey J. Management of spontaneous pneumothorax: British Thoracic Society pleural disease guideline 2010. Thorax. 2010;65(Suppl 2):ii18–ii31.
  2. Advanced Trauma Life Support, 9th ed. 
  3. Inaba K, Ives C, McClure K, Branco BC, Eckstein M, ShatzD, Martin MJ, Reddy S, Demetriades D. Radiologic evaluation of alternative sites for needle decompression of tension pneumothorax. Arch Surg. 2012;147(9): 813Y818.
  4. Inaba K, Branco BC, Eckstein M, Shatz DV, Martin MJ, Green DJ, Noguchi TT, Demetriades D. Optimal positioning for emergent needle thoracostomy: a cadaver-based study. JTrauma. 2011;71(5):1099Y1103; discussion 103.
  5. Laan D V., Vu TDN, Thiels CA, et al. Chest wall thickness and decompression failure: A systematic review and meta-analysis comparing anatomic locations in needle thoracostomy. Injury. 2015:14–16. 
  6.  Laan D, Berns KS, Habermann EB. Needle thoracostomy: Clinical effectiveness is improved using a longer angiocatheter. 2015. doi:10.1097/TA.0000000000000889.
  7. Hecker M, Hegenscheid K, Völzke H, et al. Needle decompression of tension pneumothorax. J Trauma Acute Care Surg. 2016;80(1):119–124. doi:10.1097/TA.0000000000000878.
  8. Carter TE, et al. Needle Decompression in Appalachia Do Obese Patients Need Longer Needles? West J Emerg Med, 2013; 14(6): 650–2
  9. Clemency BM, Tanski CT, Rosenberg M, May PR, Consiglio JD, Lindstrom HA. Sufficient catheter length for pneumothorax needle decompression: a meta-analysis. Prehospital and disaster medicine. 30(3):249-53. 2015.
  10. Ferrie EP, Collum N, McGovern S. The right place in the right space? Awareness of site for needle thoracocentesis. Emerg Med J. 2005;22(11):788–789.
  11. Inaba K, Karamanos E, Skiada D, et al. Cadaveric comparison of the optimal site for needle decompression of tension pneumothorax by prehospital care providers.

GCS: Saying What We Mean

The Gist:  The Glasgow Coma Scale (GCS) is widely used, yet complicated by clunkiness and poor inter-rater reliability (explanation of kappa).  The Simplified Motor Score (SMS) is easier to use and equivalent, although this is prone to similar limitations.  Until a better means of communicating mental status comes, it may be best to communicate what the patient is doing (opening eyes to voice, moaning incomprehensibly, localizing pain). See this ScanCrit post.

The Case: A 29 year old male involved in a MVC with multiple traumatic injuries resulting in a prolonged ICU course at Janus General had a tracheostomy placed for respiratory failure.  The patient was responding appropriately to questions, following commands, opening his eyes spontaneous and lacked any signs of confusion or delirium, mouthing words, but was awaiting tracheostomy exchange for a fenestrated trach with a Passy-Muir valve.  What's the patient's GCS? Does this patient's GCS reflect their mental status?
  • Documented at as 10NT, 11T, and 15 by various providers.  The arguments behind each: 10NT - cannot test verbal, 11T -one point for showing up, 15 -patient oriented, and saying appropriate words, just without phonation.
In medicine, we communicate through abbreviations, codes, and numbers. When we see heart rate or blood pressure values we can place these numbers in the context of our knowledge of the patient’s peers. These numbers become actionable.  

Other critical components to the physical exam and evaluation are less easily quantified.  The mental status, for example, is a key component to evaluating a patient.  The GCS was developed to communicate the mental status of a head injured individual among providers during continuing care in a neurosurgical unit [1].  It is often used to track neurologic status when transferring care or over time.  A particular GCS in the prehospital setting may also qualify a patient for a trauma activation in some settings. 

Limitations: Unfortunately, unlike other vital signs, scores don't have explicit meaning.  The total GCS is often reported, yet this 13 point scale (3-15) actually has 120 different possible combinations.  A patient with a GCS of 10 may be completely oriented but totally paralyzed or be moaning incomprehensibly with their eyes own and a withdrawal reflex present.  Further, the sum of the GCS does not equal the parts, with regard to mortality. Healey et al used the National Trauma Database to model mortality predictions based on GCS and found that the same total sum score could be associated with double the mortality (ex: from 27% to 52%) depending on the individual components. Further, the mortality associated with scores is not linear [3].  So a GCS of 11, for instance may mean very different things for two different patients.

Yet, even if the numbers did mean something, the GCS has been found to have abysmal inter-rater reliability.  In one study, 19 emergency physicians rated 131 patients within five minutes of each other found a concordant GCS 32% of the time (Spearson's rho 75, weighted kappa 0.40) [4].   Even in the rather protected setting of case based written scenarios, emergency providers the overall GCS accuracy was 33.1% (95% CI, 30.2-36.0) [5].  In a written mock scenario, EMS personnel (n=178) generated an accurate GCS one-quarter of time without a scoring aid and a shocking 57% with the use of a scoring aid [7].

Alternatives: Given the inaccuracy of the GCS, Thompson et al set out to determine whether the performance of the SMS, a truncated version of the GCS was equivalent to the GCS in a retrospective cohort of out of hospital head injured patients. In the SMS, points are awarded for obeying commands (2), localizing pain (1), and withdrawing to pain or worse (0). They found that the predictive nature of the SMS paralleled that of the GCS, although the GCS seemed to predict mortality slightly better,  0.90 using GCS (0.88-0.01) vs 0.87 using SMS (0.86-0.88) [7].

So, what do we do?
Trashing the GCS is simply not an option for most of us; yet, score cards don't seem to do us any favors.  For example, during a trauma activation, the expectation (at least at Janus General) is to communicate to the room the patient’s GCS.  This may seem to convey more neurologic information than the actual exam discriminates. Recognizing the limitations to the GCS is important in discerning both what we do with this information and how we communicate what we mean, whether it's in documentation, to other providers, or family members.

  • Describe the exam.  With the knowledge of the subjectivity and poor reliability of the GCS, one may give the breakdown of points rather than a simple total GCS and describe the neurologic examination.  Documenting descriptors in medical records this may aid other teams in tracking the patient's exam.  
  • We may also engage in interdisciplinary discussions about use of simplified scoring systems such as the SMS or about the ways we communicate and document neurologic exams. 
References:
1. Green SM. Cheerio, laddie! Bidding farewell to the Glasgow Coma Scale. Ann Emerg Med. 2011;58(5):427–30. doi:10.1016/j.annemergmed.2011.06.009.
2. Singh B, Murad MH, Prokop LJ, et al. Meta-analysis of Glasgow Coma Scale and Simplified Motor Score in predicting traumatic brain injury outcomes. 2013;27(March):293–300. 
3. Healey C, Osler TM, Rogers FB, et al. Improving the Glasgow Coma Scale score: motor score alone is a better predictor. J Trauma. 2003;54(4):671–678; discussion 678–680. 
4. Beveridge R, Ducharme J, Janes L, Beaulieu S, Walter S. Interrater reliability of Glasgow Coma Scale scores in the emergency department. Ann Emerg Med. 2004;43(February):215–223. 
5. Bledsoe BE, Casey MJ, Feldman J, et al. Glasgow Coma Scale Scoring is Often Inaccurate. Prehosp Disaster Med. 2014. doi:10.1017/S1049023X14001289.
6. Feldman A, Hart KW, Lindsell CJ et al. Randomized controlled trial of a scoring aid to improve glascow coma scale scoring by emergency medical services providers. Ann Emerg Med. 2015 Mar;65(3):325-329.e2.
7. Thompson DO, Hurtado TR, Liao MM, Byyny RL, Gravitz C, Haukoos JS. Validation of the Simplified Motor Score in the out-of-hospital setting for the prediction of outcomes after traumatic brain injury. Ann Emerg Med. 2011;58(5):417–25.