Trying to create a rollupAve of a number, where it doesn't count fields that are empty?

I've created a calculated field where I've successfully created a rollupAve function to a related module:

rollupAve($contacts_rtc_session_2,"avg_speaker_score_c")

This works great, but I'd like for it NOT to include in the calculation any instances where the field "avg_speaker_score_c" is empty. Is this possible? If so what do I need to change the formula to?