Hopes have been raised in recent weeks that the pair may eventually get in the ring after six years of waiting and wanting from an expectant public.
The two fighters met for the first time last Tuesday when they spoke at the half-time interval of the Miami Heat vs Milwaukee Bucks NBA game.
May 2 has been penciled in as the date the fight could take place, with negotiations having reportedly stalled and time running out, Khan feels that the match-up is doomed.
"Honestly that fight will never happen, in my opinion," Khan told BT Sport.
"I spoke to Manny Pacquiao and, obviously, we speak to Mayweather and his team.
"I can't really see that fight from happening, there is too much politics with that fight. There's different promotions teams and different management teams – I don't think they will let that fight happen.
"If it does happen, I'm a big boxing fan and I would love to see it.
"I want either one of those guys, if they are fighting each other then I can't argue because I would love to see that fight like everyone else
"It's a fight I've always waited for but it's taken so long that I can't see it happening."