Drug Baron

Twenty six of the best candidate protein biomarkers have no ability to predict Alzheimer’s Disease

July 8, 2014 by admin | 24 Comments

DrugBaron woke up this morning to the latest chapter in the “blood test for Alzheimer’s Disease” soap opera, with this article from the BBC trumpeting the latest study to be published as a “major step forward”.

The study behind the headlines was published (unusually for a “major breakthrough”) in the journal Alzheimer’s & Dementia, and described a study based on the measurement of 26 candidate protein biomarkers in blood samples from 1,148 individuals, 476 with Alzheimer’s Disease, 220 with mild cognitive impairment (MCI), and 452 controls of similar age but no dementia.

The authors of the study, and their industrial collaborators at Proteome Sciences plc, were not backward at promoting this work as a significant: Dr Ian Pike, Chief Operating Officer at Proteome Science declared “Having a protein test is really a major step forwards.  [It] will take several years and need many more patients before we can be certain these tests are suitable for routine clinical use, that process can start fairly quickly now.”

Even independent voices were quick to praise the new research: Eric Karran, Director of Research at Alzheimer’s Research UK, described the study as a “technical tour de force”.  Really?

This is, after all, not the first 2014 paper to make such a claim: in March a paper in Nature Medicine made almost identical claims, and BBC article reporting that study even used many of the same stock images!  Even the headline claims of the two studies were similar (90% accuracy for the Nature Medicine paper verus 87% accuracy for the new study).  And both used similar methodology: multivariate signatures, although the earlier study was focused on metabolic biomarkers and the new study on proteins.

So did the new study justify the hype any more than the previous attempts to solve this important problem? DrugBaron reviewed the primary publication with interest.

Lets start with the positive news: the analytical components of the study seem to be largely in order.  The multiplexed ELISA assays employed are not particularly well validated, but in our experience they yield measurements that correlate with robust uniplex assays (although often only in terms of rank order of levels, as opposed to absolute levels).  There is no reason to assume, therefore, that the table of 26 measures in 1,148 blood samples are not a reasonable snapshot of the relative levels of these small sample of blood proteins in those people.

The clinical definition of the subjects is also extensive and, as far as one can tell, robust.  Its worth noting that only a minority had either rate of cognitive decline determined (n=342) or a brain MRI scan (n=476), so the ‘headline’ study size of over 1,000 only really applies to the crude classifications on the basis of medical history.

Its also worth noting that the study cohort was pooled from three separate studies.  While, generally, that’s unlikely to create a spurious positive finding, it does introduce a lot of noise into the measurements (we know that some, at least, of these analytes are very sensitive to the precise methodology used to prepare the blood samples), and so decreases the chance of finding anything useful.  However, an important control is not reported: the fraction of the subjects with MCI or AD in the three separate sub-cohorts is not described.  This can create a massive artifact, in the same way that including subjects of different races can undermine a GWAS study.  Imagine that more of the subjects from ANIM had AD than from KHP-DCR (two of the sub-cohorts used); if differences in the blood sampling and sample handling in ANIM (or KHP-DCR) caused a difference in the level of an analyte, that analyte would now (artifactually) be found associated with AD.

This subtle problem, however, is the least of the concerns

It is with the statistical analysis of the dataset the real problems begin.  The impressive-sounding statistical terminology belies some serious flaws.

Lets begin with the simplest possible analysis: univariate (that is, one variable at a time) comparison of the markers between the three symptomatic groups.   Only two of the 26 biomarkers were associated with dementia, and one of those (apoE) has been known as a key associate of Alzheimer’s Disease since the 1980s.

The authors did not attempt to determine the extent to which the new information in their apoE and CFH measures added to a predictive model of dementia status based only on APOE genotype and age.

Instead, they tortured the data

Since “many” of the measured biomarkers were associated with APOE genotype (a very strong associate with dementia status in this cohort as in most others ever studied), they corrected the biomarker levels for APOE geneotype and then performed partial correlations between the corrected biomarkers and a range of measures of brain atrophy.   With lots of biomarkers and lots of MRI measures, not surprisingly, there were a lot of “significant” correlations (there are 25 in Table 2).  They tested 364 correlations, so on average you would expect 18 significant to the level of p<0.05 by chance alone.  Consistent with that, they mention, briefly, that only two of the correlations survived correction for multiple testing.  One of those was ApoE again.  Its quite unclear what value Table 2 serves other than to confuse readers unclear about the perils of multiple testing.

At this point, they should have concluded that there is little extra information in their dataset beyond APOE genotype (and maybe apoE levels).

Undeterred, however, they used multiple linear regression on what is really quite a small dataset and “identified” a signature associated with brain atrophy.   The problem with this methodology is that there is no external prediction to assess the generalizability of the model.  That variation in six markers “explained” (rather than “predicted”, as they state) 19.5% of the variability in hippocampal volume (their selected measure of brain atrophy) is true within the 476 people for whom they had MRI data but is likely to be a chance finding unique to this cohort.

The associations between the biomarkers and rate of cognitive decline suffered from the same flaws (the univariate correlations were not corrected for multiple testing, and the general linear model was not tested for generalizability).

The solution is to use multivariate statistics with a test set and a hold-out set, something they did for their prediction of conversion from MCI to AD, but inexplicably failed to do for the general linear models associating biomarkers and brain atrophy or cognitive decline.  Without the external prediction test of generalizablility, these conclusions carry little or no value.  One can only assume the generalizability tests were negative.

Then we come to the piece de resistance: comparing the subset of 51 MCI patients who went on to develop AD in the next year or so with 169 MCI patients who did not.  This time, they correctly divided the group into a training set, used to build the model, and a test (or hold-out) set used to test the generalizability of the resulting model.

Unfortunately, the significance of the predictions made in this test set are not reported, and because the test set is now so small (with only 12 or 13 MCI converters to predict the presence of) even an ostensibly quite good prediction rate need not be significant.  Its unclear why the p values (and indeed confidence intervals) associated with the AUC of the Receiver Operator Curves are not given, but one is once again left assuming its because they are not significant (particularly after multiple correction, since they assembled a dozen different ROC curves, even before they started playing with the cut-offs).

Of even greater concern, the prediction of conversion to dementia based on APOE genotype alone is indistinguishable from the model using the protein biomarkers.  There is no evidence whatsoever for additional value in the protein biomarkers compared to the well-established predictive marker of APOE genotypes.  Even if there were a visible difference in the power of the models, there is no power in such a small test-set to ask whether there is really additional predictive information present in the protein biomarker data.

“There all all sorts of headlines today about how there’s going to be a simple blood test for Alzheimer’s soon. Don’t believe them” – Derek Lowe

During the day, there has been some criticism of the hype surrounding this study based on the limited use of a test with 87% accuracy.  In reality, to be useful clinically a 90% accuracy is insufficient for population screening, but it could be useful to select patients for admission to clinical trials or for monitoring response to treatment.  It would certainly be an advance over what is currently available, and for that reason along would be welcome progress.

But sadly, this study does not deliver a test with such properties.  It does not deliver any evidence that the biomarkers they examined contribute any independent diagnostic or prognostic information beyond the simple APOE genotype test that has been known for years.  A more accurate title for the paper would be: “Twenty six of the best candidate protein biomarkers have no ability to predict conversion to dementia from prodromal disease beyond the information contained in the APOE genotype test”.  That paper, however, may not have been accepted for publication even in Alzheimer’s and Dementia.

Far from being a “technical tour de force”, this study joins a long line of illustrious predecessors that have attempted to create a multivariate biomarker signature to predict disease, including the study published in March in Nature Medicine, which made the same claims while suffering from many similar methodological flaws.

Such is the complexity of multivariate statistics that its unsurprising such studies defeat the ability of journalists to critique them.  The BBC, and other news organizations, can hardly be castigated for running “breakthrough” stories.  The blame lies with a peer-review system that can allow such blatant flaws to go uncorrected, and with “experts” such as Eric Karren from Alzheimer’s Research UK who should know better before offering an independent opinion on such research, but perhaps most of all with the authors, such as Ian Pike, who, perhaps hopeful of commercial gain, made claims that are completely unsupported by their data.

 

 

 

 

 

 

  • Baron’s agenda?

    ‘Drug Baron’, you say that 18 of the correlations would have been found to be significant on chance alone, but you perhaps miss the history and wider picture. Markers such as Complement Factor H, a2-macroglobulin and clusterin have found as connected to AD in several other studies. Their occurrence here is clearly therefore not by chance as you suggest.

    I’ve read many of the articles about this work and I was surprised by how heavily caveated they are, stating clearly that this test will only be used for trial work in the near-term and that a general use test may be 2-5 years off. You barely acknowledge that, instead making out it’s all spin and hype.

    What’s your agenda?

    • davidgrainger

      My “agenda”, if it is not already sufficiently transparent, is to highlight flawed science that risks misleading those with less experience in interpreting multivariate statistics (which is a specialist task).

      I do not “miss the history” – the candidate biomarkers were indeed selected on the basis of pre-existing data linking them to AD. That does not change the limitations of the statistical design of the current study. My point is that if the data had been generated by a computer random number generator, rather than with a multiplex ELISA, you would have got (on average) 18 ‘significant’ correlations (and getting 25, as the authors did, would be no surprise). Hence, the published data is consistent with being random numbers, not meaningfully associated with AD.

      Nothing in the paper suggests that the biomarkers measured are useful at all in predicting AD, or future conversion from MCI to AD. The danger is that the flawed conclusions drawn without proper statistical tests will mislead people into wasting their time and money doing more trials with these biomarkers when this study shows very nicely that they have NO value.

      We cannot blame the authors for all the hype – although some (and the Proteome Science authors seem most to blame) have contributed public statements that underline the over-interpretation of the data (you might more profitably ask what is their agenda?). But we can blame them (and the reviewers who recommended publication) for such a poor quality scientific study. They should know better.

      In the end, I suspect I share the same passion and desire as you do to see a test for AD become available. Where we differ is that you see publications such as this one as a helpful step on the way. Maybe not “the answer” but at least an encouraging signpost – for which the authors should be encouraged, and perhaps funded further to investigate if their test really does have value. I, on the other hand, see it as clear evidence of their inability to contribute meaningfully to this field, and indeed a signpost in completely the wrong direction – one that runs the risk of getting well-meaning people to support experiments whose outcome is predictable (in failure). For those of us whoi really want to see advance in AD, the only option is to counter incompetence and hype with carefully argued critique.

      If that agenda seems unfair or unreasonable, you can stick to the mainstream press who, through ignorance, follow YOUR agenda.

      • Baron’s agenda?

        Yes markers were initially selected, but importantly, they were then tested. The panel of 10 improved the predictive power over ApoE alone.

        Now, you can fairly say that the data set was not necessarily big enough to prove that positive findings are real, but at the same time it certainly doesn’t prove there is no benefit as you claim.

        Please don’t take this the wrong way, but your logic is flawed. If finding 25 markers to correlate is consistent with the ‘at random’ number you claim of 18, that fits, but it doesn’t prove they’re not meaningfully associated!
        The results show that the panel performed with predictive ability. Your random set would not do so, so your comments about artifactual association with AD become void.

        ‘Nothing in the paper suggests that the biomarkers measured are useful at all in predicting AD’

        They tested the predictive power and found at the 85%
        sensitivity, the test achieved an accuracy of 87% with a
        specificity of 88%. Does that or does that not suggest the markers are useful in predicting AD?
        Why make out there’s nothing in the paper that suggests utility in prediction when clearly there is.

        The benefit of results like that is that pharma companies will be able to run trials with far fewer patients, making the testing of new drug candidates far more practical (and cheaper)

        Let’s face it, your article is rather one-sided. The press releases have all been realistic in terms of caveats and timeframes, but you’ve all out attacked them with cherry-picked and misleading remarks.
        Clearly you’ve just wanted to cause a stir and a bit of hype, to help get attention to your blog and TCP

        My agenda is integrity and honesty. Please don’t try to suggest I’m a sucker for anything the mainstream press say

        • davidgrainger

          There is no point repeating the arguments made in the original post. But there is no flaw in my logic.

          The studies show that the associations between the biomarker panel and the disease markers are no greater than would be expected from chance alone (beyond the APOE genotype, apoE and apoJ levels at any rate). That is neither “encouraging” nor progress.

          However, the argument will be settled by future studies. Despite my (well-founded) skepticism, Im sure the authors will attempt to test their panel in a prospective study (recruit subjects, run the test and predict who will convert from MCI to AD and then watch and see who actually does). That result will prove whether there is anything useful here or not. I can be pretty confident that this will not demonstrate anything like 87% accuracy!

          The sense in which the way the study was promoted is harmful is that it risks deflecting resources from trying other approaches that MIGHT be successful into testing this panel which is already shown to be not usefully better than random.

          • Baron’s agenda

            ‘The studies show that the associations between the biomarker panel and the disease markers are no greater than would be expected from chance alone ‘

            The NUMBER of correlations is no greater than chance alone, sure. However, the PERFORMANCE of the correlated markers IS encouraging whether you like it or not. In that respect, the number of correlations becomes rather irrelevant – indeed, the fact the correlated proteins are mostly known to have a role in AD, or have been confirmed in other studies, shows that chance did not come into it. Something to bear in mind that why you say chance would generate 18 on average, the vast majority of events in real life fall away from average.

            Backed up by a) the independent studies re several of these markers that you clearly haven’t bothered to look at and b) the sister study that used different analytical approaches, the markers become more interesting still.

            You say that the performance was dire, yet now say you’re confident the markers wont achieve ‘anything like 87% accuracy!’
            If they performed so badly before, then 87% accuracy would be poor and easy to achieve again, right? It seems you consider 87% accuracy bad when you want to make out the panel achieved nothing, but good when you want to suggest they’ll not hit that level again. Make your mind up.

            You say funds should be directed to other approaches that ‘MIGHT be successful’ to suggest this one has no hope.
            A test that achieves 87% accuracy, albeit in a smallish study, does not suggest no hope. Based on your logic, no drug at all should be progressed even with very positive Phase II studies.

            ————————————————-

            The fact is, this set of markers performed well in terms of predicting progression from MCI to AD.
            What you can say is that the study wasn’t large in all respects and could have been distorted by certain aspects and so MIGHT NOT replicate in a larger study – but you seem unable or unwilling to acknowledge that it MIGHT DO (and like it or not, the statistics suggest a good chance of that)

          • actual_statistician

            No, it is not. You could randomize the outcomes and this routine (take the top K univariate markers) would generate the same results. This is also a way to demonstrate better-than-chance performance for routines like this, and unsurprisingly the authors did not do this.

      • lukas10

        ‘Where we differ is that you see publications such as this one as a helpful step on the way. Maybe not “the answer” but at least an encouraging signpost – for which the authors should be encouraged, and perhaps funded further to investigate if their test really does have value. I, on the other hand, see it as clear evidence of their inability to contribute meaningfully to this field, and indeed a signpost in completely the wrong direction – one that runs the risk of getting well-meaning people to support experiments whose outcome is predictable (in failure). For those of us whoi really want to see advance in AD, the only option is to counter incompetence and hype with carefully argued critique’

        David – it is unfortunate that you don’t apply statistical rigour to your own remarks.
        You cannot honestly say that this study is not an encouraging signpost, or that experiments based on the test can only result in failure.
        The results DO if anything show a possible role in the test panel in aiding prediction.

    • John

      Baron’s agenda?, I hesitate to jump into the middle of the squabble here, but there is something you’ve made reference to twice now in your arguments that I don’t quite understand and I ‘m hoping you can elaborate. Above, you wrote, “Markers such as Complement Factor H, a2-macroglobulin and clusterin have found as connected to AD in several other studies. Their occurrence here is clearly therefore not by chance as you suggest” and you reiterated something similar by stating that, “you’ve repeatedly avoided acknowledging that the observed proteins fit well with the sets found in other discovery studies and are therefore not just found by chance.”

      To me, these statements make it sound as if your argument is that the work must be valid, in part, because out of a sea of random proteins, the researchers’ work specifically identified proteins that have been previously connected with AD. Is that what you’re stating? I mention this only because, as I understand the paper, that is in direct contrast to what the researchers actually did; they did not start with a sea of random proteins, but rather started with 26 proteins that were already known to be connected with AD. Therefore, any identified protein biomarker in their study will already have–by definition, based on its inclusion in their analysis–a connection to AD.

      And for the record, I have no agenda whatsoever here. I will say, however, that–like David–I don’t think the researchers ever should have continued once they showed ApoE was driving the bus. After that, it just looks like they fiddled around with the numbers until they found something they liked; it reminds me of doing post-hoc analyses on everything you can imagine until you find something that’s significant and then conveniently forgetting about everything you did that was not significant.

  • Kelvin

    The number of potential correlations with 26 variables îs 2^26 = 67,108,864, which is way, WAY higher than the number of samples tested (only a few hundred). You’re virtually *guaranteed* to find some correlation with those numbers, so the chances that the correlation is both meaningful and reproducible are virtually NIL.

    • Kelvin

      PS. They may as well have looked at hair colour, and make of car, etc., rather than at these pseudo-scientific genetic markers. It would have been just as useless but far cheaper to gather the information.

      • davidgrainger

        The markers are neither pseudo-scientific nor genetic, by the way! Lets keep the criticism to the justifiable flaws…

    • davidgrainger

      Kelvin – be careful. They did not test all possible inter-correlations between 26 variables. They tested univariate correlations between each variable (26) and seven MRI parameters (7) in two groups – MCI and AD (2) = 364 actual tests made. Of these, 5% will, by chance, be “significant” (364/20) = 18ish.

      This multiple testing problem is the reason that they then adopted multivariate models (linear regression and feature selection in their case). These failed because they didn’t test the generalisability of the general linear models, and they didn’t test the significance of the predictions made with the selected features in the hold-out set (or rather they didn’t report these tests – they very probably are negative by eye).

      These kind of flaws be-devil these multivariate biomarker signatures. But the tools do exist to do these problems, and a study of this size is large enough if (a) there really was any predictive power in the chosen markers; and (b) optimal statistical methodology was rigorously employed, Sadly, neither applies in this case.

  • bank

    Quite a scathing review, I must say. Though I’m surprised that Ian Pike was singled-out for criticism rather that the senior author of the manuscript.

    • lukas10

      Agreed. Proteome Sciences obviously turned David down for some consultancy work, so he’s trying to put the boot in!!

      • davidgrainger

        For absolute transparency, lets be clear: I had never even heard of Proteome Sciences until yesterday, The review is scathing because the science is flawed – but in a rather technical way that will likely not be obvious to most non-scientists (and indeed even to many scientists who do not specialise in multivariate statistics).

        There is no axe to grind here. Just a desire for clarity and technical excellence.

        • lukas10

          David
          So you were CEO of Fingerprint Diagnostics Ltd and then Director and CSO of Prognostics Ltd, working on blood-based multiplex diagnostics and a focus on Alzheimer’s, but claim not to have heard of Proteome Sciences before yesterday?

          Even the most basic assessment of the competitive space in which you operated would have brought up the company. It’s not like there are many others about. Your companies overlapped in Alzheimer’s, proteomics, blood diagnostics and multiplex approaches!

          Please don’t take this the wrong way, but are you telling porkies or was your market knowledge that limited while running those companies?

          • davidgrainger

            Happy to be charged with limited knowledge of the competitive landscape. Were Proteome Sciences around back then?

            Neither FingerPrint nor Pronostics ever even looked at Alzheimer’s let alone had a “focus” on it, as you suggest. FingerPrint was entirely focused on cardiovascular disease, and Pronostics launched multiplexed products in the autoimmune space.

            But then again, none of this history is relevant. This is not an attack on Proteome Sciences or any other party. It is simply my independent scientific critique of their publication. Readers can judge for themselves what they believe.

          • lukas10

            You may want to get your bio changed. Presumably you authorised this?

            “FingerPrint Diagnostics Ltd (who develop novel multivariate diagnostic tests for diseases such Alzheimer’s Disease or atherosclerosis)”
            http://www.praxiscourses.org.uk/about/David_Grainger.asp

            It also seems your old CEO thought there was a focus on AD.

            Maybe that’s why the company went bust?

            “Booth said: “We are tremendously excited by the potential released through the merging of two technologies at the forefront of medical diagnostics.

            “We have worked successfully with FPD for over two years now but this merging of interests will broaden and accelerate the availability of a new and revolutionary class of medical diagnostics, ‘pronostics’, based on biological profiling, which will benefit patients and health care providers worldwide.

            “Together, we will deliver an entirely new type of diagnostic which will provide rapid and simple non-invasive diagnosis of diseases such as heart disease, Alzheimer’s and cancer, enabling improved patient care and outcomes.””

            ———————

            ‘Readers can judge for themselves what they believe’

            yes, which is why I felt it was important to add a bit of context to your remarks.

          • davidgrainger

            Should anyone still be reading this lengthy exchange, we may as well have the facts correct for the record (even though, as we have both said, my ‘history’ isn’t really relevant to the objectively verifiable scientific critique I posted).

            FingerPrint developed a method of obtaining a deep, multivariate “profile” of the immune-system (an ‘immunomics’ platform). The vision (which Pronostics also shared when they acquired FingerPrint) was that such a data-dense profile might be useful to define many (or every) disease an individual might suffer in the future. Such a hope for multivariate diagnostic profiles was quite common a decade ago (which is the period we are talking about).

            FingerPrint worked on the proof-of-principle test and characterised its performance in healthy subjects over time, and investigated whether the profile was associated with heart disease in two very large clinical studies. It was not (well, had we applied the same statistical methodology as the authors of the Alzheimer’s and Dementia paper, it would have APPEARED to be useful – but that would have been flawed too),

            So, despite general statements that such tests might be useful in Alzheimer’s as well as a host of other chronic diseases, neither company ever spent a penny investigating Alzheimer’s Disease. The general statements you quote are not inconsistent with my assertion that we did not work on Alzheimer’s Disease in either company.

            The company ultimately failed precisely because we faced up to the reality that the biomarker signatures did NOT predict the disease we studied. We did not use inferior methodology and the complexity of multivariate statistics to kid anyone (including ourselves) that we had something useful. It is that history that makes it irksome to see an equal failure (albeit in a different disease) misrepresented as useful. If you wish to describe that as “bitterness” thats your view.

            But what it does do is give me an unusually relevant technical understanding of the area to offer an in-depth scientific critique of the current study methods – an expertise, I would suggest, that no-one could have without being historically “involved” in the area (something you seem so concerned about as being a potential ‘conflict of interest’).

            Its five years since I worked in this area for Pronostics. I think thats long enough to make me as independent as anyone else with a sufficiently deep understanding of the complex methods used. You cannot undermine a scientifically-valid critique by raising concerns about an “agenda”.

          • Baron’s agenda?

            ‘the biomarker signatures did NOT predict the disease we studied’

            Which is clearly why you remain bitter
            While you say it was five years ago, it’s nonsense to pretend that’s long enough to make you independent.
            You spent years working on these approaches… they failed and the company went bust.
            To claim you’re not bitter about that is a joke.

            You still pretend to have undertaken a rigorous review here, but you simply have not!
            As I have shown, your logic is off, resulting in your whole argument falling down.

            1. You take overlap with numbers generated by chance as PROOF the observed proteins must be worthless
            a) while interesting, it’s no proof whatsoever and you know it
            b) you’ve repeatedly avoided acknowledging that the observed proteins fit well with the sets found in other discovery studies and are therefore not just found by chance
            c) you’ve similarly avoided acknowledging that these proteins have been validated in independent studies as well as showing connection to pathways known to be involved in AD

            But MORE IMPORTANTLY, you’re deliberately trying to make out that flaws in how the markers were identified must mean the results are worthless.

            1. In many cases, the discovery flaws you highlight would only have made it HARDER to achieve good levels of prediction

            2. The study found the marker set to have good predictive value regarding MCI to AD conversion in a year.
            Yes or no David?
            The answer as you know is yes.
            Now, for reasons you highlight, those results may not be the most robust, but they are the results nevertheless.

            IF the method was flawed, then the teams may not be able to achieve such good values in a follow-on study, but to suggest there’s no chance they will is nonsense as you know full well.
            Like I’ve said, the flaws in the test method are overcome to quite a degree by the support from other research.

            To make out there’s NO chance of success in a larger study and that it’s a total waste of money to try is just DISHONEST distortion David.
            It’s a shame you couldn’t have written a fair and balanced review devoid of the bitterness from your past failures.

          • Lukas10

            It seems we’re now getting somewhere

            In 2001 David, you set up Fingerprint Diagnostics and were its CEO
            Your two primary areas of focus were Alzheimer’s disease or atherosclerosis.

            That merged with another firm, to create Prognostics Ltd where you were its Chief Scientific Officer and a director (as CSO, your role was presumably to assess the market and the science?). The focus moved somewhat.

            In 2009, Prognostics Ltd was liquidated due to a lack of operational funding. The assets were picked up by Phadia

            — So, it seems rather likely you’re bitter about seeing 10 years of your work, focused on AD, go down the pan due to lack of funding, while Proteome Sciences and it’s world-leading set of collaborators is getting somewhere. Is that why you picked on Ian Pike?

            Oh look, here’s Prognostics Ltd and Proteome mentioned alongside each other in the British in vitro diagnostics association annual report.
            Most odd that you say you’ve never heard of a company that would have been your primary competitor for years!!

            http://www.bivda.co.uk/Portals/0/Documents/Annual%20Review/Bivda_Annual%20Review.pdf

            I have nothing against you David – you’re clearly a smart chap – but please let’s not let your history cloud what could otherwise have been a thoughtful piece of analysis.

          • davidgrainger

            Your mis-representation of my so-called “agenda” here is laughable. But you are entitled to your opinion, which is here for everyone to read and form their own judgement.

            An “agenda” might affect opinion, or spin, but this piece is based on an objectively verifiable criticism of statistical methods. The science is flawed, whether or not I am motivated by “bitterness” as you suggest.

          • Baron’s agenda

            ‘this piece is based on an objectively verifiable criticism of statistical methods’

            you make out it is, but you mislead and distort as has been shown.
            you draw conclusions that cannot be fairly drawn in the ways that you suggest

            You make out the identified markers must be junk just because probability says around that number might be returned on average by luck. Without proper recognition that these are markers showing up time and time again in independent work, you write them off.

            But it goes beyond that. You’re claiming there was no data to support the markers having predictive ability, but such prediction is exactly what the paper shows!

            You’re not making honest remarks. You’re dressing up distorted misrepresentations in snippets of statistics to try and make them sound authentic. Instead of acknowledging what the study reported and then commenting on that, you twist it.

            As I’ve said, you can by all means say you don’t think the encouraging results (and they can only be described as encouraging) will be replicated in further studies, but that’s an opinion not a fact.

            To dismiss the panel outright in the way you have goes against all of the statistical and scientific approaches you like to think you follow.

            Good results from a small study, or a less than completely ideal design does not mean success cannot be the end outcome. So why do you pretend otherwise.

            Statistics are one thing, but joined with flimsy logic they become rather dangerous as you’ve shown!

            I’ll leave it at that. Validation through a contract with pharma to use this panel in a drug trial will rather support my views.

  • Baron’s agenda?

    ‘The studies show that the associations between the biomarker panel and the disease markers are no greater than would be expected from chance alone (beyond the APOE genotype, apoE and apoJ levels at any rate). That is neither “encouraging” nor progress’

    Your logic IS flawed
    Do you not see – there are two parts of it
    1. showing markers to be associated
    2. testing the predictive ability of those markers

    You’re muddling things up!!

    - chance is not reality, but in any case, as I’ve said, it wouldn’t have mattered if the first part found only 5 markers to be associated.
    You’re rubbishing the whole piece on an aspect that is not directly relevant.

    - it is the PERFORMANCE of the markers that is encouraging and progress.
    a) the markers found were clearly not the result of chance as indicated by their roles in AD and other supporting evidence
    b) the panel achieved 87% accuracy

    Perhaps you’ll answer this:
    ”””””””””””””””””””””””””””””’
    i) would 18 markers found purely by chance achieve 87% accuracy?

    ii) would many of these same markers have shown up in other discovery projects using different sample sets if it was down to chance?

    iii) is the result of 87% more robust when one considers that an analysis using different technology returned comparable results?

    iv) based on the things the panel DID achieve, not the things it didn’t, and taking into account all of the other research relating to these markers, is it fair to claim it has no chance at all of achieving success in a separate large study as you have?