After working my way through Chapter 2, I think the analysis is as follows.

First, we look for a "saddle point" (probably the same as the Nash equilibrium flagamuffin mentioned).

Each side assumes the other side will play optimally. So if Red plays 1, Red assumes Blue will play 2 and win five instead of three. And if Red plays 2, Red assumes Blue will play 1 and win six instead of four. Red would rather pay five than six, so Red's preferred move is 1 with a value (loss) of five.

Blue faces the prospect of winning only three with move 1 versus winning at least four with move 2. So Blue prefers 2 to win at least four.

Red's best worst case is five, Blue's best worst case is four. These don't match, so we conclude that a "pure" strategy (always making the same move) is not optimal for either side.

Next, we determine the best mixed strategy for each side. With move 1, Red is exposed to a difference of two in the possibilities for Blue. With move 2, Red is again exposed to a difference of two depending on how Blue plays. So Red doesn't prefer either move, and should play both moves with equal frequency. Now Red doesn't care how Blue plays. Whenever Blue plays 1, Red will pay 3 or 6 with equal frequency, averaging 4.5. Whenever Blue plays 2, Red will pay 5 or 4, again averaging 4.5.

Determining Blue's strategy should confirm the value. With move 1, Blue is exposed to a difference of three depending on what Red does. With move 2, the difference in outcomes is one. So Blue's best strategy is to prefer move 2 at a ratio of three-to-one. Whenever Red plays 1, Blue wins (1x3 + 3x5) / 4 = 4.5. Whenever Red plays 2, Blue wins (1x6 + 3x4) / 4 = 4.5.

So Red plays at 1:1 and Blue plays at 1:3 and the value of the game is 4.5.