Some people are no good for you, no matter how you try to convince yourself.
If nothing lights up in them when they are with you, nothing ever will.
You have one life to live and you simply cannot allow people with whom you have no resonance to becloud your journey. Get rid of them.