In July 2012, we launched a computational Challenge, The DREAM-Phil Bowen ALS Prediction Prize4Life Challenge, with Prize4Life to better predict the progression of disease in ALS patients. Earlier this month, Prize4Life announced the winners. The judging panel received an overwhelming response, with 1,073 Solvers having signed up, and submissions coming from around the world. Given the quality of the submissions, the judging panel doubled the original prize purse to $50,000. We’re very pleased to have Dr. Neta Zach, Scientific Director for Prize4Life, join us to discuss this Challenge. (Ed Note: Dr. Zach’s colleague, Dr. Melanie Leitner, was interviewed in July at the launch of the Challenge).
Hello Dr. Zach – Could you take us back to the beginning and help us to understand what motivated you to run this Challenge, and in particular, why you opted for a computational “Big Data” Challenge?
Prize4Life’s mission is to accelerate the development of treatments and a cure for ALS. We have embraced the prize-for-breakthrough model in part because we are interested in attracting new and innovative ideas to ALS research. One area that we identified with great potential is quantitative analysis of ALS data. To that end, we developed the PRO-ACT (Pooled Resource Open-Access ALS Clinical Trials) database in collaboration with the Northeast ALS Consortium (NEALS) and the Neurological Clinical Research Institute at MGH, with funding from the ALS Therapy Alliance. PRO-ACT, which will be launched in early December, contains information from over 8,500 patients from past clinical trials, ten times more than had been previously available.
The DREAM-Phil Bowen ALS Prediction Prize4Life Challenge (a.k.a. ALS Prediction Prize) was a way to utilize, for the first time, the new and promising PRO-ACT database. Specifically, we wanted to use it in order to confront a basic puzzling question in ALS: most patients are like Lou Gehrig, with a rapidly progressing disease course. Some patients, however, turn out to be more like Stephen Hawking, where the disease progression is delayed. What separates the Lou Gehrigs from the Stephen Hawkings? Understanding the variability of the disease can mean a lot for ALS patients going through diagnosis and can lead to a substantial reduction in the cost of clinical trials for ALS treatments. The unique approach of providing ALS “Big Data” to a global community of researchers speeds up the process while driving down the costs of discovery, which is good news for both the scientific and patient communities we serve.
Having run a couple of high-profile Challenges now, was there anything that particularly surprised you during the course of this Challenge?
We were happily surprised by the level of engagement this prize received. We had over a 1,000 registered Solvers. You can see the engagement in the quality of the winning solutions, but beyond that, the fact that so many Solvers – from 64 countries no less – were actively engaged on the forum and through emails, with over 300 forum messages and 1,500 emails, speaks to the success of this Challenge! Indeed, this engagement bore fruit – even the solutions that didn’t win were so valuable that we encouraged several of the Solvers to submit for publication in scientific journals to share their algorithms with the ALS community at large. This was our first large scale interaction with the quantitative community and we were thrilled by their hard work and devotion to the cause.
I understand that two teams secured first place, each winning $20,000, and a second place winner was awarded $10,000. In fact, the judges decided to double the size of the prize purse based on these submissions. What stood out about these submissions and differentiated them from the others?
The winning Solvers of the ALS Prediction Prize developed algorithms that predict a given patient’s disease status within a year’s time based on three months of data. We saw an overwhelming response to the competition, and the efforts from data scientists worldwide resulted in at least three viable (and potentially complementary) solutions. Among the many proposed solutions submitted over the InnoCentive "Prodigy" platform, those offered by the two first place teams scored virtually identically (and the best!), even though both the statistical methods and the parameters chosen by each team were different. In addition, the solution offered by the second place winner scored extremely close to that of the other two teams.
The winning algorithms were also deemed to be of great merit by our clinical judges looking at the ability to implement them for ALS patient care and to develop new avenues of ALS research. Given the quality of the results submitted, our judging panel realized it was impossible to award just one prize as we had originally planned. With the help of a generous donor deeply committed to helping find a cure for ALS, we decided to double the prize amount we had initially allocated.
Can you tell us more about your plans to apply the winning solutions? Specifically, how do you think the solutions will get you closer to your goal of developing effective treatments and a cure for ALS?
We are currently assessing ways in which the algorithms could benefit day-to-day clinical practice as well as using them in a clinical trials context. We already know that the solutions to the ALS Prediction Prize will have two important and immediate benefits: they will increase the likelihood of clinical trial success, and our experts estimate that these algorithms can reduce the number of patients in a clinical trial by 23%.
We are also working with a number of the Solver teams to ensure that the winning solutions, as well as some of the best performing non-winning algorithms, will be published in scientific journals, so that the results will be as broadly available as possible for the ALS community at large.
You have a big announcement coming up in December surrounding the PRO-ACT database. Could you tell us more about PRO-ACT and provide us with a peek into what you’re going to announce?
Yes! The PRO-ACT database, which provided the data for this Challenge, will be open to the public on December 5, 2012. The launch of this database will be an opportunity for the ALS Prediction Prize winning and non-winning Solvers, and other researchers from around the world, to ask (and hopefully answer) many other interesting questions about ALS progression, stratification, response to medications, and more. We believe that the ALS Prediction Prize is just the tip of the iceberg when it comes to the promise that the PRO-ACT database holds for fostering treatments and a cure for ALS, and we and our many partners are happy to share the data with the world in just a few short weeks.
Our Solvers always value feedback – what words of wisdom can you provide to them as they work on and attempt to solve future Challenges?
First of all, I would like to note how grateful we are to all of the Solvers. Sometimes this goes unsaid, but I want to state it loudly and clearly: your bright minds and new and innovative ideas are the reason for the success of this Challenge. Thank you!
There is something else I want to a share with the InnoCentive Solvers. One thing we saw in the Challenge is that some Solvers who developed algorithms appeared to give up, and despite all their hard work, did not submit a full report to be judged. We believe that when some Solvers saw that other Solvers were scoring better on the leaderboard, they were discouraged. In the end, we actually discovered that the best algorithms on the leaderboard are not necessarily the best overall. So, I want to encourage all of the registered Solvers to please not despair – your algorithm might have been better than you believed. For future Challenges, if you worked hard and you have an algorithm that runs (or a general solution for a non-computational Challenge), you have nothing to lose by submitting it for evaluation!