Your Mutual fund order may get rejected for the following reasons: 

  1. Insufficient balance in your bank account for the order to be placed.
  2. Attempting to place a new Mutual fund order on the same day as selling your MF holdings may result in rejection, as the fund settlement (T+3) has not yet been completed.  
  3. If the order is rejected due to any issues at the AMC's end, the funds will be refunded to your bank account, and this process may take up to 10 working days. 

Related articles:

what is the mutual fund settlement cycle
how long will it take for my mf units redemption amount to reflect in my bank account