Derivation of the Gibbs entropy formula from the Boltzmann entropy

There are many ways to arrive at the Gibbs entropy formula $S= -k_B \displaystyle \sum_{i} p_i \log p_i~,$

which is actually identical, except for units, to the Shannon entropy formula. Here I document a relatively straightforward derivation of the formula which might be the easiest route to develop intuition for undergraduate students who already know the formula for the Boltzmann entropy, $S(E)= k_B \log \Omega(E)~,$

where $\Omega(E)$ is the effective number of configurations of an isolated system with total energy $E$. This derivation is not new of course, but I decided to write it up anyway because it is a particularly elegant argument and is not always given emphasis in the usual textbooks.

Let us start by considering some general statistical ensemble such that a state $i$ has energy $E_i$ and probability $p_i=p(E_i)$ which is the same for all states with the same energy $E_i$. For fixed energy $E_i$ we can assume, from the principle of equal a priori probabilities, which is valid in the microcanonical ensemble, that $p_i = \displaystyle \frac 1 {\Omega(E_i)}~.$

Hence we obtain for the Boltzmann entropy $S(E_i)=k_B \log \Omega(E_i) = -k_B \log p_i~.$

For our general ensemble, the mean entropy is given by the weighted average, over all possible states, of the Boltzmann entropy, according to $S= \langle S(E_i) \rangle _i = \sum_i p_i S(E_i) = \sum_i p_i~ (-k_B \log p_i)$

from which we obtain the famous Gibbs formula for the Boltzmann-Gibbs entropy: $S = -k_B \displaystyle \sum_i p_i \log p_i~.$