Preheat oven to 350 F. Grease and line a 9 inch round cake pan with parchment paper.
Cream sugar with unsalted butter until fluffy, using a stand mixer with the paddle attachment, or in a large bowl using a hand mixer. Cream for about 10 minutes on high speed. Add vanilla essence and almond extract.
Crack eggs in a separate medium bowl and whisk gently with a fork until frothy, about 7-8 minutes. Add the orange and lemon zests, as well as the milk.
Add egg mixture to the butter and sugar mixture, a little at a time.
Sift the dry ingredients (flour, baking powder, cinnamon, nutmeg, salt) into the batter. Beat the flour mixture on into the butter mixture on low speed to combine.
Pour cake batter into prepared pans. Place cakes in the middle rack of the preheated oven, and bake 35-40 minutes until a toothpick inserted in the center of the cake comes out clean.
Remove the cake to a wire rack to cool for 10 minutes, then run a thin knife around the sides of the pan to cleanly remove the cake from the pan.
Notes
This cake keeps wrapped very tightly in plastic wrap for 1-2 days.