A US appeals court has posed tough questions at those challenging and defending President Donald Trump's controversial travel ban.