When Jesus walked by Levi the tax collector and invited him to follow, what did the other disciples do? Can you see Peter turning up his nose and generating reasons why Jesus was wrong for inviting a tax collector to follow him? Can you imagine Andrew wondering if the people he introduced to Jesus are going to be angry with him now because Jesus reached out to someone who was outside their comfort zone? I think it was only because these guys were so new at walking with Jesus that they didn't have the courage or the ability to object to Jesus calling Levi.
I once worked for a church where the leaders told me that I was reaching the "wrong kind of kids." I was reaching out to any kids who would come to activities. They wanted me to reach out to kids from respectable families whose parents would attend church services with them. I can only imagine how these leaders would have reacted to Jesus calling Levi if they had been there.
One truth about following Jesus is that he calls all kinds of people, and those who are following do not get to choose who those people will be. This is either the worst imaginable news, or the greatest opportunity - it all depends on if you are thinking like Jesus or just thinking about yourself.