A much-touted behavioral therapy for autism, the Early Start Denver Model (ESDM), may not be as effective as its creators had hoped1.
In the latest study of the therapy, it did not improve children’s intelligence quotients (IQ) or adaptive behavior any more than other treatments. Children treated with ESDM showed some improvement in their language, but only at two of the three study sites.
Independent experts say the results are disappointing, and they question some of the methods used to generate them.
“There are questions to be answered about exactly how the results were generated,” says Jonathan Green, professor of child and adolescent psychiatry at the University of Manchester in the United Kingdom, who was not involved in the study. “The way the trial is reported is not totally transparent.”
In ESDM, parents or caregivers use the child’s interests to practice several skills at the same time. For example, they might harness a child’s interest in playing with blocks to work on her imitation, language and motor skills.
Several therapies require less intensive effort yet produce better results, Green says.
For example, the Preschool Autism Communication Trial (PACT) — which Green is involved with — and JASPER each require only a fraction of the time needed for ESDM. PACT has been shown to deliver sustained improvements in autism traits and JASPER benefits language skills.
“Given the overall context of autism intervention science at the moment, [the new results] are extremely disappointing,” Green says.
ESDM’s creators take issue with this interpretation of the findings.
“There’s certainly more than one high-quality treatment out there, but to me it says [ESDM] is one of them,” says Sally Rogers, professor of psychiatry and behavioral sciences at the University of California, Davis MIND Institute, who led the trial. “It may not be better than what kids are getting in the community, but they consistently make, as a group, excellent gains.”
No guarantees:
ESDM showed great promise in 2010, when Rogers and her colleagues reported that it increased IQ, improved adaptive behavior and eased core autism traits in 24 autistic children more than a combination of standard treatments did.
A follow-up study in 2015 suggested that the therapy has lasting effects. But critics at the time noted that the study included only 39 children, most of whom come from affluent white families, and that many of the assessments were based on parent reports.
The new trial included 118 children aged 14 to 24 months at three sites; the children were randomly assigned to have either ESDM or whatever local therapies their parents chose to sign them up for. Therapies available in the community included applied behavioral analysis, occupational therapy and language therapy. The parents knew which group their child was assigned to, but the assessors did not.
For two years, children received a weekly average of 17 hours of ESDM or 14 hours of other treatments.
At the start of the trial, and after one and two years of treatment, the researchers evaluated the children’s language, motor skills and visual perception using the Mullen Scales of Early Learning. They also assessed adaptive behavior based on parent interviews, and autism severity using the Autism Diagnostic Observation Scale.
Only 81 children completed the trial; 10 of 55 in the ESDM group and 27 of the 63 controls dropped out. Green and others say this level of attrition is unusually high.
Averaged across all three sites, the data suggest that ESDM improves language skills more than the other treatments do. But this effect is not apparent in data from one of the sites. The work appeared in January in the Journal of the American Academy of Child & Adolescent Psychiatry.
“On average, we have differences between the [control and ESDM treatment] groups,” says Gerhard Hellemann, assistant professor of statistics at the University of California, Los Angeles, who led the data analysis, but “we can’t guarantee that every single site has the average effect.”
Some independent experts, however, are not convinced.
The trial did not replicate the improvements in Mullen scores and in adaptive behavior that “led most people to be excited about the original paper,” says Jeremy Veenstra-VanderWeele, professor of psychiatry at Columbia University, who was not involved in the research.
“The majority of previous findings were not replicated,” says Connie Kasari, professor of human development and psychology at the University of California, Los Angeles, who was not involved in the work.
High bar:
The researchers first presented the results at a 2014 conference. They reported that ESDM improved children’s IQ and language at two of the three sites, at odds with the new report.
The results changed because colleagues questioned the statistical analysis in the conference report, Rogers says.
“The first analysis never went to press, was never submitted, because it was not done correctly,” she says. “Every single piece of the analysis that was done at that time has been redone. It’s not the same statistician.”
Some experts say the new statistical methods are difficult to interpret, however. Rogers says that her team also changed the primary outcome measure, the specific improvement that the researchers had hoped the trial would show. “In the 2014 presentation, we did specify a primary variable for the analysis and reported on it. It was [developmental quotient],” an overall measure of the child’s development. The new paper’s primary outcome measure is language.
Experts say changing the primary outcome measure is problematic.
“A lot of trials have been reported in the past with multiple outcomes, and then people have picked and chosen what outcome they wanted depending on what looked most positive,” Green says. “That’s called post hoc reporting, and it’s what shouldn’t be done.”
Still, taking the results at face value, ESDM may have shown little effect in the trial in part because the controls benefited from their therapies significantly more than did those in the 2010 study. This is a sign that therapies have improved over time, says Elizabeth Berry-Kravis, professor of pediatrics at Rush University in Chicago, who was not involved in the work.
“The bar is higher for this study,” she says.
Some of the therapists treating the controls may have used methods inspired by ESDM — which could also depress the apparent effects of the therapy. “I talk to therapists all the time who say, ‘Oh yeah, we don’t officially do ESDM, but we know what that methodology is and we incorporate that type of thing in our therapy,’” Berry-Kravis says.
References:
- Rogers S.J. et al. J. Am. Acad. Child Adolesc. Psychiatry Epub ahead of print (2019) PubMed