Side fat can be very stubborn and hard to get rid of. The reason I used the word stubborn is because that is exactly what it is, at least for men. When we exercise, our bodies don't release fat equally from all areas. For men, the area where fat is released slowly is the lower/side abs, for women it is the hips/pelvic. That makes losing fat from those areas more difficult.
Performing exercises, aerobically and anaerobically, that target your oblique can significantly help in that direction.