[Q] Contacts, show contacts from only certain groups AND those unassigned. topic

I have three groups Family, Friends and Cooworkers.
Most of my contacts fall in a group above but some in none (eg. my barber) and I don't want to over-organize things.

I want the contacts/dialer to show all contacts, except from group Cooworkers.

Any ideas on how to achieve this?

1) I've tried setting the filter to Family, Friends - and sure Cooworker don't show but neither do unassigned.
2) If I include the group "My Contacts" - EVERYTHING is shown, regardless of other filters (how is that a feature?)

If I go with 1) then I have to remember to add new contacts to a group and frankly, I don't trust myself to do that.
So that's no option.

BEST would be if there'd be another group called "Unassigned" and I could then just tick Family, Friends, Unassigned...

How do I achieve this functionality?
Any other app capable of this?
Ideas?