Place the cubed challah in a greased 9"x 13"x 2" baking dish.
In a large bowl, combine the milk, brown sugar, eggs, vanilla, cinnamon, cardamom, ginger, allspice, nutmeg, and salt. Pour over the challah, making sure that the bread is completely covered by the milk mixture. Sprinkle the almonds on top.
Cover and refrigerate for at least an hour or overnight.
Set out the chilled bread pudding while you preheat the oven to 350°F (175°C).
Bake 40 to 45 minutes, or until a knife inserted into the center of the pudding comes out clean.
To make the vanilla sauce:
Place the butter, sugar, brown sugar, cream, and salt in a heavy saucepan*. Cook over medium heat until bubbly and thickened (about 5-8 minutes).
Remove the pan from the heat. Carefully stir in the vanilla.
Spoon over servings of warm bread pudding.
Notes
*The mixture will be fairly bubbly while it cooks, so be sure to use a large enough pan.
Vanilla sauce recipe slightly adapted from Land O'Lakes.