The greedy baker with a full barrel and a drunken wife

Earlier this week, I gave a talk at one of the UCL Priment seminars. The session was organised around two presentations aimed at discussing the definition and differences between "confidence" and "credible" intervals $-$ I think more generally, the point was perhaps to explore a little more the two approaches to statistical inference. Priment is a clinical trial unit and the statisticians and clinicians involved there mostly (if not almost always) use frequentist methods, but I think there was some genuine interest in the application of Bayesian inference. I enjoyed very much the event. 

For my talk, I prepared a few slides to briefly show what I think are the main differences in the two approaches, mostly in terms of the fact that while in a Bayesian context inference is performed with a "bottom-up" approach to quantify and evaluate $$\Pr(\mbox{parameters / hypotheses} \mid \mbox{observed data})$$ a frequentist setting operates in the opposite direction using a "top-down" approach to quantify and evaluate $$\Pr(\mbox{observed data / even more extreme data} \mid \mbox{hypothesised parameters}).$$
I think that went down well and at the end we got into a lively discussion. In particular, the other speaker (who was championing confidence intervals), suggested that when the two approaches produce results that are numerically equivalent, then in effect even the confidence interval can be interpreted as to mean something about where the parameter is. 

I actually disagree with that. Consider a very simple example, where a new drug is tested and the main outcome is whether patients are free from symptoms within a certain amount of time. You may have data for this in the form of $y \sim \mbox{Binomial}(\theta,n)$ and say, for the sake of argument that you've observed $y=16$ and $n=20$. So the 95% confidence interval is obtained (approximately) as $$\frac{y}{n} \pm 2\,\mbox{ese}\!\left(\frac{y}{n}\right),$$ where ese$(\cdot)$ indicates the estimated standard error for the relevant summary statistics. 

The probabilistic interpretation of this confidence interval is 
$$\Pr\left(a \leq \frac{y}{n} \leq b \mid \theta = \frac{y}{n} \right) = 0.95$$
and what it means is that if you're able to replicate the experiment under the same conditions, say 100 times, then 95 of them the resulting confidence interval would be included between the lower and upper limits $a$ and $b$. With the data at hand, this translates numerically to 
$$\Pr\left(0.6211\leq \frac{y}{n} \leq 0.9788 \mid \theta=0.80\right) = 0.95$$
and the temptation for the greedy frequentist is to interpret this as a statement about the likely location of the parameter.

The trouble with this is that by definition, a confidence interval is a probabilistic statement about sampling variability in the data $-$ not about uncertainty in the parameters as shown in the graph below.

(in this case, in fact 4 out of 100 hypothetical sample means fall outside the confidence interval limits, but this is just random variation).

The "alternative" interpretation, rescales the graph above by using the sample mean to produce the one below.

While this is perfectly valid from a mathematical point of view, however, this interpretation kind of muddies the water, I think, because it kind of confuses the point that this is about sampling variability in the data. And the data you would get to see in replications of the current experiment would be about the observed number of patients free from symptoms, $y^{new} \sim \mbox{Binomial}(\theta=0.8,n=20)$. 

Thus, it is a bit confusing, I think, to say that a Bayesian analysis could yield the same numerical results as the confidence interval $-$ perhaps a more appropriate interpretation would be that this is a probabilistic statement in the form 
$$ \Pr\left(na \leq y \leq nb \mid \theta=\frac{y}{n}\right) = 0.95, $$
which in the present case translates to 
$$ \Pr\left(12.43 \leq y \leq 19.58 \mid \theta=0.8\right) = 0.95, $$
which suggest we should expect to see a number of patients free from symptoms between 12 and 20 in replications of the current experiment (which is of course in line with the first graph shown above, on the scale of the observable data).

The Bayesian counterpart to this analysis may assume a very vague prior on $\theta$, for example $\theta \sim \mbox{Beta}(\alpha=0.1,\beta=0.1)$. This would suggest we're encoding our knowledge about the true probability of being symptoms-free in terms of a thought experiment where $\alpha+\beta=0.2$ patients were observed and only $\alpha=0.1$ were symptoms free (ie a prior estimate of 0.5 with very large variance).

Combining this with the data actually observed $y=16, n=20$, produces a posterior distribution $\theta\mid y \sim \mbox{Beta}(\alpha^*=\alpha+y, \beta^*=n+\beta-y)$, or in this case $\theta \mid y \sim \mbox{Beta}(16.01,4.1)$, as depicted below.

A 95% "credible" interval can be computed analytically and as it turns out it is $[0.6015; 0.9372]$, so indeed very close to the numerical limits obtained above when rescaling to the sample mean $y/n$ $-$ incidentally:
1. It is possible to include even less information in the prior and so get even closer to the frequentist analysis;
2. However, even in the presence of large data that probably will overwhelm the prior anyway, is it necessary to be soooo vague? Can't we do a bit better and introduce whatever prior knowledge we have (which probably will separate out the two analyses)?
3. I don't really like (personally) the term "credible" interval. I think it's actually un-helpful and it would suffice to call this an "interval estimate", which would highlight the fact that given the full posterior distribution, then all sorts of summaries could be taken.

The Bayesian analysis tells us something about the unobservable population parameter given the data we've actually observed. We could predict new instances of the experiment too, by computing
$$ p(y^{new}\mid \theta,y) = \int p(y^{new}\mid \theta)p(\theta\mid y) \mbox{d}\theta. $$
This means assuming that the new data have the same data generating process as those already observed and that we propagate the revised, current uncertainty in the parameter to the new data collection. In some sense, this is even the "true" objective of a Bayesian analysis, because parameters typically don't "exist" and they are just a convenient mathematical feature we use to model observable data.

The confidence interval can make probabilistic statements about the likely range of observable data in similar experiments that we may get to see in the future. The Bayesian analysis tells us something about the likely range of the population parameter, given the evidence observed and our prior knowledge, formally included in the analysis. 

While I think the construction of the confidence interval has some meaning when interpreted in terms of replications of the experiment (particularly with respect to the actual data!), I believe that the stretch to the interpretation as a probability for the parameter is a unwarranted, hence the reference to the greedy baker who wants to have and eat his cake $-$ which in Italian would translate as having a full barrel (presumably of wine) and a drunken wife...

So I guess that, in conclusion, if you enjoy a tipsy partner, you should be Bayesian about it...

Coming soon!

We've just received a picture of the cover of the BCEA book, which is really, really close to being finally published!

I did mention this in a few other posts (for example here and here) and it has been in fact a rather long process, so much so that I have made a point of joking in my talks about BCEA that we'd be publishing the book in 2036 $-$ so we're in fact nearly 20 years early...

The official book website is actually already online and I'll prepare another one (I know, I know $-$ it's empty for now!), where we'll put all the relevant material (examples, code, etc).

I think this may be very helpful for practitioners and our ambition is to make the use of BCEA as wide as possible among health economists $-$ even those who do not currently use R. The final chapter of the book also presents our BCEAweb application, which can do (almost everything!) that the actual package can (the nice thing about it is that the computational engine is stored on a remote server and so the user does not even need to have R installed on their machine).

We'll probably have to make this part of the next edition of our summer school...

Three rooms left...

Last December, Kobi and his classmates did their Christmas play, which was based on a relatively close representation of the Nativity (well $-$ perhaps back then shepherds used to run around with most of their hands up their nose, waiving at their parents too...).

Anyway, one of the top acts of the whole thing was something like this, hence the title of the post.

But, more importantly, we're almost running out of single rooms for our summer school, later this year, in Florence (although there are more double rooms)! So book your space soon!

Face value

This is actually a not-so-recent paper, but I've only discovered now and I think it's very interesting. The underlying issue is about trying to do "causal inference" from observational data $-$ perhaps one could see this in a simpler way by considering the idea of "balancing" observational data, to mimic as far as possible an experimental setting (and so be able to estimate "causal" effects). [There's lot more on the philosophical aspects behind this problem, which I'm conveniently swiping under the carpet, here...]

Anyway, one of the most popular ways of dealing with this issue of unbalanced background covariates (or generally, confounding) is to use propensity score matching. But, while I think that the idea is somewhat neat and clearly important, what has always bothered me (among other things) is the fact that the resulting outcome model does assume that the estimate of the propensity score (PS) is "perfect" $-$ known with absolute precision, although the basic assumption is that "the PS model needs to be correct". But of course, there's no way of knowing perfectly that the PS model is correct...

So the idea of joining model selection and propagation of uncertainty through the outcome model is actually very interesting. I've only flipped through the paper and I did have some very preliminary ideas in mind on this, so I really want to have a proper look at this!

New year resolution

Now that the Christmas break is just a distant memory (Marta would say that I am quite happy with that $-$ she thinks I'm like the Grinch around the Christmas holiday. And she is right), I've given way to my new year's resolution of finally, properly packaging our two R packages that aren't on CRAN yet.

The first one is SWSamp (about which I've already talked here and here) and the second is survHE (which I have also already mentioned here and here).

I've got better at using GitHub and (for survHE) benefited from the help of Peter Konings, who's helped with bits of code and also given me either tips or "forced" me to look into better solutions for the management and potential distribution of the packages, even if they're not directly on the official R repository.

Eventually, this means I've settled for (I think!) a good compromise $-$ I've created a local repository in which I've stored my packages; this in itself doesn't take care of all the dependencies, but it's easy (even for practitioners not too familiar with R) to install the packages and all the others on which they rely to work with very simple commands, for example

install.packages("survHE",
    repos=c("http://www.statistica.it/gianluca/R",
        "https://cran.rstudio.org",
        "https://www.math.ntnu.no/inla/R/stable"),
    dependencies=TRUE
)

$-$ this way R uses three repositories (one for survHE, one for all the other dependencies stored on CRAN and one for INLA, which is under its own repository).

We've done some tests and all seems to be working OK, which is great. I've also set in motion a couple of plans for updates to both the packages $-$ I'll post more on this soon! (Incidentally, this also gives way for the development of two more interesting projects: Anthony's work on single arm trial and Andrea's work on missing data for cost-effectiveness analysis. Again, will post more as we have some more output to show for!).

Movie stars

Our search for potential alternatives to an academic career, in the face of increasing competition and difficulties in securing grant money has now led Jolene, Marcos and me to seek employment in the show-biz $-$ just in case we fail to recruit enough students for our new MSc in Health Economics & Decision Science...

PhD opportunity!

Applications are invited for a PhD funding opportunity to conduct research in a branch of probability or statistics based in the UCL Department of Statistical Science, commencing in September 2017. This funding is provided by the Engineering and Physical Sciences Research Council (EPSRC).

The requirement for admission to the MPhil/PhD in Statistical Science is a 1st class or high upper 2nd class Bachelor’s degree, or a Master’s degree with merit or distinction, in Mathematics, Statistics, Computer Science, or a related quantitative discipline. Overseas qualifications of an equivalent standard are also acceptable. Further details can be found on the Departmental website. Applicants are expected to prepare an outline proposal of their work. We have some interesting project in our pipeline, including extensions of our work on survHE, or related to evidence synthesis and network meta-analysis, as well as the use of observational data for health economic evaluation.

The studentship will be four years in duration and covers tuition fees at the UK/EU rate plus a stipend of £16,785 per annum for eligible UK residents. EU nationals who have not been ordinarily resident in the UK for 3 years prior to the start of the studentship may still qualify for a fees only award. The studentship may only be awarded to applicants liable to pay tuition fees at the UK/EU rate (i.e. it cannot be used to part-cover overseas tuition fees). 


Further information, including details of how to apply, is available here.

Bayes 2017

We've just opened the call for abstract for the next edition of the Bayes Workshop $-$ this time we're going to Spain and to be more precise to Albacete.

The format is the same as in the past few years $-$ you can send your abstract (including title, authors and not exceeding 300 words) at info@bayes-pharma.org. We're pretty much open to many research areas, as long as they involve Bayesian statistics (I feel I have to say this $-$ in the past we had invariably at least a couple of abstracts that had absolutely nothing to do with a Bayesian analysis!...).