Email Sync... but only those contacts Sugar likes :-)

We're having a random issue with the Outlook Connector.  It connects between Sugar and Outlook with no issues, but will only sync certain emails.  The email addresses are available within the CRM to provide the link and the correct contact appears in the side bar but we have to manually archive the emails that it doesn't sync, which defeats the point of having the connector in the first place.  

Any ideas on what this could be? 

Many thanks