Shalt thou not remove - This was spoken before the Babylonish captivity; and here is a promise that if they will return from their idolatry, they shall not be led into captivity. So, even that positively threatened judgment would have been averted had they returned to the Lord.
Return - The repentance of Israel described in Jeremiah 3:21-25 was a hope, and not a reality. The return, literally, would be their restoration to their land; spiritually, their abandoning their sins.
Jeremiah 4:1-2 should be translated as follows:
If thou wouldst return, O Israel, saith Yahweh.
Unto Me thou shalt return:
And if thou wouldst remove thy abominations from before Me,
And not wander to and fro,
But wouldst swear truly, uprightly; and justly
By the living Yahweh;
Then shall the pagan bless themselves -
In him - In Yahweh. Two great truths are taught in this verse;
(1) that the Gentiles were to be members of the Church of the Messiah;
(2) that Israel‘s special office was to be God‘s mediator in this great work.
Thus, Jeremiah is in exact accord with the evangelical teaching of Isaiah.