This is because there is a difference between your skin undertones and that of the powder. You much choose a shade that completely complements your skin tone. Otherwise lighter powder tone will appear as ashy and white on darker skin tones. Make sure you choose the right color.
Marilynn is right. You have to make sure that the color isn't too light for your skin tone. You should go to a high end beauty store.. they have people there that can do your makeup and show you exactly what to buy.
Do you use a moisturizer before putting on the minerals? I notice that if I don't do that, then it does look ashy. A trick to not have it be ashy is to lightly mist your face with water after you've put on your minerals. It kind of helps set it.
Also, if using minerals, you're probably using a veil to set it. Try to use a veil that is close to the same color as your foundation. Hope this is somewhat helpful.