Had Kaworu met someone else who was vulnerable, he would have fallen for him instead. Any lilin would have done, he just wanted to love someone. Shinji is replaceable to him which is why he easily dumps him for Kaji or Rei in Rebuild. Same with Rei, had someone else been kind to her first she wouldn't have cared about Shinji either.
This goes on with Asuka, Misato, etc. all the characters so it's a fruitless argument.
Shinji on the other hand liked Kaworu because he was his ideal, which isn't so easily recreated with other people.