Kenneth G. Wilson (1923). The Columbia Guide to Standard American English. 1993.
buff (n.), in the buff
Buff was originally the yellowish leather of the American buffalo or bison, and it came to refer to both the leather, its color, and the leather-covered block of wood used to polish or shine. Buff then came to mean skin-colored, and skin: hence the Conversational idiom in the buff, meaning naked. The word buff meaning fan is Informal and Conversational, as in Im an opera buff, and comes apparently from the nineteenth-century enthusiasts who rooted for their favorite New York fire companies (the firemen wore buff-colored coats).