This guy is rejected by everybody. He asked me out, I said no and now I feel bad?

Think about it, their must be a reason why everybody is rejecting him. Try your hardest to ignore people if they do start rumors or something, if that does not work start a rumor yourself or blend in with the crowd and take their attraction and turn it somewhere else.