Gay-rights advocates filed a court challenge Thursday to the government's unusual plan to canvass Australians' opinion on gay marriage next month, while a retired judge said he would boycott the survey as unacceptable.

The mail ballot is not binding, but the conservative government won't legislate the issue without it. If most Australians say "no," the government won't allow Parliament to consider lifting the nation's ban on same-sex marriage.

Lawyers for independent lawmaker Andrew Wilkie and marriage equality advocates Shelley Argent and Felicity Marlowe, applied to the High Court for an injunction that would prevent the so-called postal plebiscite from going ahead.

"We will be arguing that by going ahead without the authorization of Parliament, the government is acting beyond its power," lawyer Jonathon Hunyor said.

Prime Minister Malcolm Turnbull said the government had legal advice that the postal ballot would withstand a court challenge.

"I encourage every Australian to exercise their right to vote on this matter. It's an important question," Turnbull said.

Gay-rights advocates and many lawmakers want Parliament to legislate marriage equality now without an opinion poll, which they see as an unjustifiable hurdle to reform.

Retired High Court judge Michael Kirby, a gay man who supports marriage equality, dismissed the ballot as "irregular and unscientific polling."

"It's just something we've never done in our constitutional arrangements of Australia, and it really is unacceptable," Kirby told Australian Broadcasting Corp.

Kirby would not comment on the legality of the government proceeding with the 122 million Australian dollar ($96 million) ballot without Parliament's approval, but said: "I'm not going to take any part in it whatsoever."

Plebiscites in Australia are referendums that don't deal with questions that change the constitution. Voting at referendums is compulsory to ensure a high voter turnout and that the legally-binding result reflects the wishes of a majority of Australians.