Liebowitz Social Anxiety Scale (LSAS): Optimal cut points for remission and response in a German sample

The Liebowitz Social Anxiety Scale (LSAS) is the most frequently used instrument to assess social anxiety disorder (SAD) in clinical research and practice. Both a self‐reported (LSAS‐SR) and a clinician‐administered (LSAS‐CA) version are available. The aim of the present study was to define optimal cut‐off (OC) scores for remission and response to treatment for the LSAS in a German sample. Data of N = 311 patients with SAD were used who had completed psychotherapeutic treatment within a multicentre randomized controlled trial. Diagnosis of SAD and reduction in symptom severity according to the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, 4th edition, served as gold standard. OCs yielding the best balance between sensitivity and specificity were determined using receiver operating characteristics. The variability of the resulting OCs was estimated by nonparametric bootstrapping. Using diagnosis of SAD (present vs. absent) as a criterion, results for remission indicated cut‐off values of 35 for the LSAS‐SR and 30 for the LSAS‐CA, with acceptable sensitivity (LSAS‐SR: .83, LSAS‐CA: .88) and specificity (LSAS‐SR: .82, LSAS‐CA: .87). For detection of response to treatment, assessed by a 1‐point reduction in the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, 4th edition, rating, a reduction of 28% for the LSAS‐SR and 29% for the LSAS‐CA yielded the best balance between sensitivity (LSAS‐SR: .75, LSAS‐CA: .83) and specificity (LSAS‐SR: .76, LSAS‐CA: .80). To our knowledge, we are the first to define cut points for the LSAS in a German sample. Overall, the cut points for remission and response corroborate previously reported cut points, now building on a broader data basis.

bootstrap; cut point; LSAS; receiver operating characteristics; social anxiety

Social anxiety disorder (SAD), formerly known as social phobia, is defined as a persistent fear of embarrassment or negative evaluation while engaged in social interaction or public performance (American Psychiatric Association, [1] ). The disorder has an early onset and a chronic course, which can result in severe psychosocial impairments and high socio‐economic costs (Keller, [23] ; Kessler, [24] ). People suffering from SAD typically fear a variety of social situations such as a meeting or interactions with strangers, attending social gatherings, or formal presentations. As secondary effects, they are less educated, less likely to marry, and more likely to be unemployed (Heimberg et al., [15] ; Kessler, [24] ; Mannuzza et al., [33] ). Furthermore, SAD is often associated with comorbid psychological disorders (e.g., major depression and alcoholism) and influences social role functioning and help‐seeking behaviour (Keller, [23] ). According to the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM‐5), the 12‐month prevalence of SAD reaches 7% in the United States (Ruscio et al., [37] ). For Germany, Wittchen, Stein, and Kessler ([49] ) reported a 12‐month prevalence rate of 5.2% and a lifetime‐prevalence between 2.5% and 7.3% (Wittchen, Essau, von Zerssen, Krieg, & Zaudig, [48] ). Despite its high prevalence, SAD is rarely diagnosed due to the difficulty in recognizing its symptoms as belonging to a disorder, as the symptoms can often be confused with individual characteristics, especially in certain cultures (Dos Santos, Loureiro, Crippa, & de Lima Osório, [7] ; Filho et al., [9] ; Gren‐Landell et al., [13] ). Therefore, the use of reliable diagnostic methods is particularly necessary. Studies comparing structured interview to unstructured diagnoses usually find the former outperforming the latter (Shear et al., [40] ; Stuart et al., [44] ). In particular, several studies have positively evaluated the reliability and validity of the Structured Clinical Interview for DSM (SCID; Lobbestael, Leurgans, & Arntz, [31] ; Ventura, Liberman, Green, Shaner, & Mintz, [46] ) for diagnoses according to DSM‐5 (First, Williams, Karg, & Spitzer, [11] ). However, application of the SCID is time‐intensive and costly, so it is not performed with every patient and if so only at admission. That is why several authors tested the utility of self‐report and clinician‐rated scales to detect SAD (Mennin et al., [35] ) and monitor treatment response (Bandelow, Baldwin, Dolberg, Andersen, & Stein, [3] ).

The Liebowitz Social Anxiety Scale (LSAS) is a well‐validated scale for the screening and assessment of SAD symptoms (Rytwinski et al., [38] ). Originally developed by Liebowitz in 1987, the LSAS uses 24 items to measure the degree of fear and avoidance experienced in a range of social and performance situations. It is available as self‐report version (LSAS‐SR) and as clinician‐administered version (LSAS‐CA; Fresco et al., [12] ). Both versions show good psychometric properties for the detection of SAD, when compared with one another and with other available scales (Osório, Crippa, & Loureiro, [36] ). They have been translated into Brazilian (Dos Santos et al., [7] ), French (Yao et al., [51] ), Hebraic (Levin, Marom, Gur, Wechter, & Hermesh, [29] ), Portuguese (Terra et al., [45] ), Spanish (Bobes et al., [4] ), Turkish (Soykan, Ozgüven, & Gençöz, [42] ), and German (Stangier & Heidenreich, [43] ). All these versions show good to excellent internal and external validity, and the scale can be considered a standard instrument in the screening and treatment evaluation for SAD, including pharmacological and psychotherapeutic trials (e.g., Mayo‐Wilson et al., [34] ) as well as cost‐effectiveness studies (e.g., Egger et al., [8] ).

Previous receiver operating characteristics (ROC) analyses of the LSAS found that the optimal cut point (OC) to differentiate between individuals with and without SAD was about 30 (Bandelow et al., [3] ; Mennin et al., [35] ; Rytwinski et al., [38] ; Santos, Loureiro, Crippa, & Osório, [39] ). However, studies in different cultural backgrounds identified cut points as low as 19.6 (Bobes et al., [4] ) and as high as 41/42 (Kummer, Cardoso, & Teixeira, [25] ) as optimal. This highlights the need for culture‐ and sample‐specific cut points and studies that try to systematically validate cut points in different samples and countries.

Another critical issue is the ability of an instrument to detect response to treatment. In clinical everyday practice, response to treatment is commonly defined as a 50% reduction from the beginning to the end of treatment on a given scale. This definition is arbitrary, as Bandelow et al. ([3] ) already mentioned, and cut‐off scores should be based on a clinically measurable improvement. In their study, they compared scores of the LSAS and the Clinical Global Impression (CGI) scale. The CGI is a global measure for disorder severity and treatment‐induced improvement in a variety of disorders. They found that patients who were “much improved” according to clinician‐rated CGI‐Improvement subscore of <2 had a mean percentage reduction in the LSAS of 31%, suggesting that the usual 50% criterion may be too conservative (Bandelow et al., [3] ). To our knowledge, this is the only published cut point for the LSAS that defined response to treatment and that was, for example, used in studies predicting treatment response in SAD (such as Hoyer et al., [20] ; Wiltink et al., [47] ).

Overall, the LSAS seems to be a valid and reliable measure for the assessment of SAD. However, either for clinical decisions or for harmonizing research definitions of response, evidence‐based cut points are strongly needed. Also, if one wants to individualize treatments based on past improvements, clear criteria for improvements are necessary. The aim of the present study was twofold: to determine valid OCs for remission for both versions of the LSAS in patients suffering from SAD and to examine the prechanges to postchanges in LSAS scores corresponding to a clinical response. For this purpose, we performed a secondary analysis for the data of an existent study with a large, reliably diagnosed sample of patients suffering from social anxiety disorder (Leichsenring et al., [26] ; Leichsenring et al., [27] , [28] ).

We defined cut points for remission and response for the Liebowitz Social Anxiety Scale (LSAS).

The cut points we found were different for the self-report and clinician administered version of the LSAS.

We recommend using the LSAS for supportive application, because it seems not capable enough to replace a diagnosis of social anxiety disorder achieved by a clinical interview.

Patients were recruited from April 11, 2007, to April 29, 2009, by the outpatient clinics of the universities of Bochum, Dresden, Göttingen, Jena, and Mainz (in Germany) as part of the SOPHONET study, a multicentre randomized controlled trial comparing the long‐term effects of psychodynamic therapy versus cognitive behavioural therapy in patients with SAD (Leichsenring et al., [26] ). For this study, a total of 1,450 potential patients were screened. The following inclusion criteria were applied (Leichsenring et al., [27] , p. 760): age range of 18–70 years; diagnosis of social anxiety disorder according to the SCID‐IV Axis I and II Disorders (Wittchen, Zaudig, & Fydrich, [50] ), LSAS score >30 (Mennin et al., [35] ), and primary diagnosis of social anxiety disorder according to the Anxiety Disorders Interview Schedule (Brown, diNardo, & Barlow, [5] ). Exclusion criteria were psychotic and acute substance‐related disorders; Clusters A (“odd, eccentric”) and B (“dramatic, emotional, erratic”) personality disorders; prominent risk of self‐harm; organic mental disorders; severe mental conditions; and concurrent psychotherapeutic or psychopharmacological treatment. Of these, 495 patients met the inclusion criteria, did not meet any exclusion criteria, provided informed consent as well as data on outcome variables, and were included in the original study (Leichsenring et al., [27] ). Finally, a total of 311 patients provided the data necessary for the secondary analyses described in the present study (see Table  for patient characteristics). The remaining 184 patients were either part of the control group (n = 79) or did not provide self‐report data at pre‐ and/or post‐assessment (n = 105). We only included patients with complete outcome data. Even though multiple imputation has some merits in effectiveness studies, it has not been tested in the context of ROC methods. All participants provided informed consent prior to inclusion in the study.

Demographic and clinical characteristics of the sample

2 Note. Data are given as means with standard deviation. LSAS = Liebowitz Social Anxiety Scale; CA = clinician‐administered; SR = self‐reported.

The LSAS (Liebowitz, [30] ; German version by von Consbruch, Stangier, & Heidenreich, [6] ) is an originally clinician‐administered, semi‐structured interview for the assessment of SAD‐related symptoms. It uses two subscales that address social interaction (11 items) and performance (13 items), measuring an individual's fear and avoidance of social situations over the past week. Answers are given on a 4‐point Likert scale. Each item is separately rated for fear (scale from 0 to 3; 0 = none, 3 = severe) and avoidance (scale from 0 to 3; 0 = never [0%], 3 = usually [67–100%]). Its total score varies from 0 to 144 points. Furthermore, the LSAS is available as a SR version, which is easier to administer and shows comparable internal consistency (Cronbach's alpha = .94) and test–retest reliability (r = .84; e.g., Fresco et al., [12] ; Levin et al., [29] ; Stangier & Heidenreich, [43] ).

The SCID‐I is a semi‐structured diagnostic interview validated for use in research and clinical settings (First, Spitzer, Gibbon, & Williams, [10] ). The interview assesses Axis I disorders listed in the DSM‐IV Text Revision and contains symptom criteria and specifiers for disorder subtype, course, and severity. Given the wide use of the SCID‐I in various clinical and research settings over several decades, it serves as the “gold standard” method to assess psychopathology (Hayden, Brown, Brennan, & O'Brien, [14] ). In the underlying study (Leichsenring et al., [26] ), 23 specifically trained and independent assessors (clinical psychologists) were masked to the treatment conditions and conducted the SCID at baseline, at Weeks 8 and 15 of treatment, at post‐treatment, and again at 6, 12, and 24 months after the end of treatment. For the current study, we evaluated only data from the baseline and post‐treatment assessment.

In line with previous studies (Mennin et al., [35] ), we defined remission as no longer meeting the SAD diagnosis at post‐therapy. Even though this classifies all patients who improved on one or more than one criteria as in remission, it provides a very direct index of clinically meaningful improvement. Response was defined as an improvement by at least one point in the Axis‐V severity rating of the SCID, which provides a global assessment of functioning. Previous studies into optimal cut points have used such global scales as anchors for cut points as well (Bandelow et al., [3] ).

Overall, four different cut points were determined; two cut points for remission (the CA and SR version) and two cut points for response (the CA and SR version). To determine a cut point score for remission, we examined which LSAS score corresponded best to either an absent or present diagnosis of SAD. To determine a cut point for response, we examined which percentage of change in the LSAS corresponded best to the criterion of clinical improvement as described above. Each of these cut points was determined in three steps. First, the OCs for the sample were calculated using ROC methodology (Bandelow et al., [3] ; Mennin et al., [35] ). For this, the sensitivity (number of remitted patients scoring below the cut point divided by the number of patients who are remitted) and specificity (number of not remitted patients scoring above the cut point divided by the number of patients who are not remitted) for all possible cut points were calculated and plotted against each other. This allows the calculation of the area under curve (AUC) value that describes the overall performance of the LSAS to correctly classify patients, irrespective of a specific cut point. Furthermore, it allows an empirically based choice for an optimal cut point. The scores that provided the best balance, that is, the smallest absolute difference, between sensitivity and specificity were defined as optimal. In addition, the positive predictive value (PPV: the probability that subjects with small LSAS scores is really remitted) and the negative predictive value (NPV: the probability that subjects with high LSAS scores is really not remitted) were reported for the optimal cut points. Second, the variability of the OCs was estimated by using a bootstrap resampling technique. Bootstrapping is a procedure to quantify the variability of an estimation by repeating the analysis several hundred times on a randomly chosen subset of participants and inspecting the distribution of results (Holländer, Sauerbrei, & Schumacher, [19] ). A nonparametric bootstrap with 1,000 repetitions was used to yield sufficiently stable estimates for the variability of the cut points. Specifically, we inspected the results for the cut points that were found in at least 5% (n = 50) of the pseudosamples. This gives some insight into the stability with which the optimal cut points can be estimated in a given sample. Earlier analyses using this method have found that some methods to define optimal cut points yield highly unreliable results (Hirschfeld & Zernikow, [17] , [18] ). The cut point that emerged as optimal in most bootstrapping samples was chosen as the OC. Third, the Reliable Change Index (RCI) was calculated to classify response as well. This method was introduced by Jacobson, Follette, and Revenstorf ([22] ) and specifies how much improvement on a given scale a patient must achieve to be classified as a responder. The RCI is equal to the individual's score before the intervention (x1) minus his or her score after the intervention (x2), then divided by the standard error of the difference (Sdiff) of the test . An absolute RCI value outside the 95% range between +1.96 and −1.96 was regarded as meaningful change (Hiller & Schindler, [16] ).

All analyses were performed in R, using the ROCR package (Sing, Sander, Beerenwinkel, & Lengauer, [41] ).

The ROC analysis indicated that LSAS‐SR scores could be used to classify patients as either remitted or not (AUC: .91; Figure ). The cut point of 30 determined in previous research achieved a sensitivity of .89 and specificity of .76. A cut point of 35 (sensitivity: .83; specificity: .82; PPV: .64; NPV: .92.) achieved the best balance between sensitivity and specificity and was met by a total of 119 (38%) patients. The analysis of the LSAS‐CA showed a similarly high level of accuracy (AUC: .94; Figure ). Here, a cut point of 30 achieved the best balance between sensitivity and specificity (sensitivity: .88; specificity: .87; PPV: .72; NPV: .95) and was met by a total of 114 (37%) patients.

The bootstrap analyses revealed that the cut points identified as optimal differed for the LSAS‐SR and LSAS‐CR (Figure ). Specifically, the majority of pseudosamples identified a cut point of 35 for the LSAS‐SR and a cut point of 30 for the LSAS‐CR (Appendix ). Further information is given in Table .

Sensitivity and specificity of the most frequently detected cut points for remission after bootstrapping

3 Note. LSAS = Liebowitz Social Anxiety Scale; CA = clinician‐administered; NPV = negative predictive value; PPV = positive predictive value; SR = self‐reported.

For the LSAS‐SR, we found overall high levels of accuracy to classify patients as either responders or nonresponders in relation to the clinical SCID rating (AUC: .85; Figure ). Furthermore, a score decrease of 28% (sensitivity: .75; specificity: .76) from the beginning to the end of treatment was identified as the OC for response, which was met by 191 (61%) patient. The LSAS‐CA achieved a similarly high level of accuracy (AUC: .90; Figure ). A score decrease of 30% (sensitivity: .79; specificity: .80) from the beginning to the end of treatment was identified as optimal and was met by 198 (64%) patients.

Bootstrapping analysis revealed a marked overlap in cut points that were identified as optimal (Figure ). For the LSAS‐SR version, a decrease of 28% from the beginning to the end of treatment was the most frequently found cut point. For the LSAS‐CA, a decrease of 29% was the most frequently found cut point (Appendix ). Further information is given in Table .

Sensitivity and specificity of the most frequently detected cut points for response after bootstrapping

4 Note. LSAS = Liebowitz Social Anxiety Scale; CA = clinician‐administered; NPV = negative predictive value; PPV = positive predictive value; SR = self‐reported.

For the LSAS‐SR, we used the test–retest reliability coefficient defined by Baker et al. ([2] ; rtt = .83) to calculate the RCI. This resulted in a RCI of 25.97. For the LSAS‐CA, we used the test–retest reliability coefficient defined by Stangier and Heidenreich ([43] ; rtt = .84). Here, a score of 24.42 represented reliable response. Agreement between RCI‐based and ROC‐based assessments was high, that is, 86% for self‐rating and 88% for clinician‐administered assessments (Appendix ).

This study aimed to determine optimal cut points for remission and response for the SR and CA version of the German LSAS, a widely used instrument for the measurement of SAD. In order to test for differences between the versions and the resulting cut points, we estimated the variability using bootstrapping. We found three main results. First, the area encountered under the curve, as well as the achieved moderate‐to‐high levels of sensitivity and specificity support the assumption that the LSAS is a useful tool for the detection of SAD. Second, the cut points yielding the best balance between sensitivity and specificity for remission were a score of 35 for the SR and a score of 30 for the CA version. Third, the optimal cut points for response were a change of 28% from the beginning to the end of treatment for the SR and a change of 29% for the CA version. In the following, we will discuss each of these findings in turn before describing general aspects concerning the variability and methods to determine cut points, as well as the limitations of the present study.

First, the overall high values are in line with previous studies. Although there are many studies showing that the LSAS can be used to detect patients with SAD (e.g., Mennin et al., [35] ; Rytwinski et al., [38] ; Santos et al., [39] ), there is essentially only one study into the utility to monitor progress during therapy (Bandelow et al., [3] ). Our findings expand this study by first showing that the LSAS can also be used in a German sample. Second, this is the first study to test both SR and CA versions. Our findings indicate that both can be used as described below different cut points should be used to interpret the results. Taken together, the present and earlier findings suggest that the LSAS is a valuable instrument in research settings to define response and remission but should not replace a clinical interview to diagnose SAD.

Second, the best balance between sensitivity and specificity was achieved for remission by using a cut point of 30 for the CA and 35 for the SR version of the German LSAS. These results are close to the cut points reported in former studies (Bandelow et al., [3] ; Mennin et al., [35] ; Rytwinski et al., [38] ; Santos et al., [39] ). We suggest using the OCs that maximize both sensitivity and specificity, because they provide the best balance between correct identification of people suffering from SAD and misclassification of those who do not. Importantly, our findings concerning the variability show for the first time that some of the differences in reported cut points may be attributed to random fluctuations, due to the estimation of the OC rather than systematic differences between the underlying samples. For example, Rytwinski et al. ([38] ) identified an OC of 30 to detect SAD, whereas the study of Santos et al. ([39] ) found an OC of 32. Both of these cut points are within the range of our bootstrapping results (see Table ). Assuming similar levels of random fluctuations across these studies, such apparently disparate findings could be harmonized.

The best balance between sensitivity and specificity for the detection of SAD by the LSAS provided an OC of 32 for a Brazilian population (Santos et al., [39] ), an OC of 19.6–26.1 for a Spanish population (Bobes et al., [4] ), and an OC of 50 for a Turkish population (Soykan et al., [42] ). Iancu et al. ([21] ) reported a high predictive correlation between female gender and higher LSAS scores. Kummer et al. ([25] ) found that the OC of the LSAS changed to 41/42 in a population of Parkinson patients. Minimally different OCs were reported by Lowengrub, Stryjer, Birger, and Iancu ([32] ), who reported OCs of 29/30 for SAD in a population of schizophrenics. These results indicate that, beside methodology, cut points may vary because of sample characteristics such as cultural background, age, or pathology.

Third, the cut points for response that we found were much lower than the conventionally used criterion of ≥50% reduction from the beginning to the end of treatment and similar to the scores defined by Bandelow et al. ([3] ). A decrease of 28% in the LSAS‐SR indicated the best detection of clinical response and a decrease of 29% in case of the LSAS‐CA. Compared with the 50% criterion, the cut points that we suggest would result in more patients being classified as responders. The changes in score from the beginning to the end of treatment were almost identical for LSAS‐SR and LSAS‐CA. In addition, we examined the response to treatment by calculating the RCI (LSAS‐SR: 25.97; LSAS‐CA: 24.42). We found substantial agreement between ROC and RCI but more participants who only showed reliable change, however without a percentage change larger than the cut point for response that we defined. This may be due to the fact that the RCI only considers the reliability of a given scale, whereas ROC analyses use clinical ratings as anchors. The average test–retest reliability of the LSAS (LSAS‐SR: rtt = .83; LSAS‐CA: rtt = .84) might result in relatively small mean changes, reaching statistically reliable improvement (Hiller & Schindler, [16] ; Jacobson et al., [22] ). According to our results, the ROC seems to be an approach more conservative than the RCI, meaning that more patients might be classified as responders by the RCI than by the ROC. For clinical practice, we recommend applying the OC of 28% for the LSAS‐SR. For the LSAS‐CA, we suggest the higher cut point of 30%, as detected by the ROC‐based analyses, to minimize the risk of overestimating treatment effects.

This study has several limitations. First, the data of the LSAS‐CA were assessed by the same examiner who assigned the SCID diagnosis. Second, in order to obtain a clinically representative sample and considering the fact that social phobia is often associated with other psychic disorders, we allowed recruitment of patients with all comorbid mental disorders less severe than SAD (according to the Anxiety Disorders Interview Schedule rating), except those listed among the exclusion criteria. The varying status and degree of comorbidity in our sample might have biased the resulting cut points, but the smaller sample size prohibited drawing firm conclusions about differences between groups. Moreover, we did not include patients with concomitant pharmacotherapy. As many patients with SAD use pharmacotherapy, this may limit generalization. Alternative cut points might be found in a population of patients treated with pharmacotherapy. Third, our sample did not include a control group, so we cannot differentiate between change due to the intervention and change due to the passage of time. Fourth, the sensitivity to treatment response of the LSAS was assessed without using a control group, which might have limited the generalizability of our cut points as well.

In conclusion, we find that the LSAS is a very useful tool to define remission and response for SAD in different clinical and research contexts. To our knowledge, we are the first who determined cut points for the LSAS in a German sample. The cut points that we found were different for the CA and SR versions of the LSAS, indicating that the use of the previously described cut points results in a larger number of misclassifications. The optimal cut points for remission were 35 (SR) and 30 (CA). For response, the optimal cut points were a percentage decrease of 28% (SR) and of 29% (CA) from the beginning to the end of treatment. The good sensitivity and specificity of the LSAS may help clinicians who do not routinely perform SCIDs at the end of treatment in order to verify whether or not patients have remitted. Also, it opens up the possibility that the LSAS may be used as an outcome measure in online studies where structured interviews cannot be performed. The strengths of the present study are the considerably large cohort of patients and the use of an internationally accepted gold standard, the SCID.

This study provides additional evidence that the cut points for both versions of the LSAS might vary. Optimal cut points are especially important given the burgeoning interest into the field of routine outcome monitoring in mental health interventions. Nonetheless, clinicians and researchers should be aware that cut points might vary due to methodology and sample characteristics.

The study was supported by a research grant from Witten/Herdecke University (IFF2014‐14).

The SOPHONET project was supported by Grant 01GV0607 from the German Federal Ministry of Education and Research (BMBF). We would like to thank all participants of the SOPHONET project. We also thank Christina Wagner, Witten/Herdecke University, for linguistic support with the manuscript.

The study was supported by a research grant from Witten/Herdecke University (IFF2014‐14).

The SOPHONET project was supported by Grant 01GV0607 from the German Federal Ministry of Education and Research (BMBF).

The study was supported by a research grant from Witten/Herdecke University.

PHOTO (COLOR): Screening/remission—receiver operating characteristic (ROC) curve and results of the bootstrapping analysis

PHOTO (COLOR): Response—receiver operating characteristic (ROC) curve for full sample and results of the bootstrapping analysis