He ate all but 6-8 babies!

[Suzie Q]

Ok, Rover had a lot (close to 100?) of fry when they hatched (hatched sometime early Monday)...All day Monday every thing was great...went to bed Monday night, he was still spitting babies out (only had one die)....Tuesday I got up, and had maybe 20 babies (started corner filter Tuesday, babies were already missing...could not find sponge filter)...by Tuesday afternoon (around 3:00pm) I only had between 6 and 8 babies left...I don't know if the filter is to strong (did I start it to early?) or if he ate them...I took him out that afternoon, and unplugged the filter over night...this morning I still have the 6/8 babies...they were not "free swimming" when I took Rover out...does it sound like he ate them, or was it the filter? I am running the filter again today, ALL the babies ARE free swimming today (only as long as I can watch it)..the current does not seem that strong. I need opinions...what is the earliest (in days of hatching) is it safe to remove the male. I had heard at around 3 days when they are free swimming. If he did eat them (I really think he did), then next spawn, should I remove him in two days instead of three? I did not feed him the first day, but started to after the babies started going missing. I did not want him eating the entire litter, so I don't think feeding him started it...he only ate a little, I think he was full by then on babies.

I really tried not to be blonde on this question, so if something does not make sense, comment and I will try to refrase it.

Additional:
Just did a count...my have 12-15 left (I can only hope!)