Most purists from the nineteenth century on have insisted that aggravate means only to make worse, as in The climate here aggravates all my symptoms, considering the annoy meaning (If that boy aggravates me much more, Im going to spank him) at best colloquial and more probably Vulgar. However, both senses seem to have been Standard all along, although Edited English uses the make worse sense almost exclusively, and you may appropriately use aggravate to mean either make worse or annoy in all but Formal contexts.