In Hatzor, the town I grew up in, there was a small mountain spring in which every Rosh ha-Shanah tashlich was done. I went there a few times myself as a child. Standing at the edge of the water, after saying a prayer that includes Micah 7:19, pockets and hems are shaken of crumbs (or just shaken, the custom varies), which symbolises sins being cast into the depths of the ocean. Hayyim Vital, Isaac kabbalist Isaac Luria’s disciple, mentioned an additional, mystical significance, that the Accuser would be cast into the depths.
The Ashkenazi custom of going on the first day of Rosh ha-Shanah, after the minhah prayer just prior to the setting of the sun, to the big sea, or to the spring, or to a well of living water, a custom termed tashlich, is a fine custom, though it is best to hold it outside the city. And a man stands on the shore, or at the edge of a well, or at a fountain and exclaims three times 'Who is a God like unto thee, &c. (Micah 7:18), Thou wilt give truth to Jacob, &c. (Micah 7:20)' which are at the end of Micah the Morashtite... And when you say 'And cast all of their sins into the sea (Micah 7:19)', focus your intentions on having all your sins and transgressions cast [into the depths] and on the Accuser on High being cast into the depths of the sea on high*, for this cause it must be said at the sea or at living waters.-Shaar ha-Kavvanot, Inyan Rosh ha-Shanah.
"עניין המנהג שנהגו האשכנזים לילך ביום ראשון דר"ה אחר תפלת המנחה מעט קודם שקיעת החמה אצל הים הגדול, או אצל המעיין, או באר מים חיים וקורים אותו תשליך, הנה הוא מנהג יפה ויותר טוב הוא אם יהיה חוץ לעיר.
ויעמוד על שפת הים, או על שפת הבאר, או המעיין ויקרא שם ג' פעמים מי א-ל כמוך כו' תתן אמת ליעקב כו' שבסוף מיכה המורשתי... וכשתאמר ותשליך במצולות ים כל חטאתם תכוון שיושלכו כל חטאתיך ועונותיך וגם המקטרג העליון יושלך במצולות הים העליון, כי לסבה זו הוצרך לאמרו על הים או על המים החיים"... (שער הכוונות עניין ר"ה).
Although entering the water is frowned upon generally, but as reported by the 19th c. Jewish traveler, Israel ben Joseph Binyamin, the Jews living in remote mountain villages of Kurdistan used to be in the habit of swimming in the water, but this died out after they emigrated to Israel.
Nobody knows when tashlich first appeared. The first literary mentions are relatively late, going back only to the late 14th century, but on the other hand, the custom is found in practically all Jewish communities, and seems to hearken back to older days. There are intriguing links with a hypothetical enthronement festival that Sigmund Mowinckel proposed took place every year between Rosh ha-Shanah and Yom Kippur. An example is the tradition in the Babylonian Talmud that kings were anointed only at a river, drawing upon David's instructions on how his son Solomon was to be made king. David Larsen mentioned a theory that the king was immersed in the Gihon at Jerusalem, which represented a death and resurrection. Another link with days gone by and the tashlich is baptism, or ritual immersion, extremely popular since at least the Second Temple period. Josephus in Antiquities 14.10 cites a decree by the city of Halicarnasus allowing Jews to build synagogues by the sea. This is not direct evidence for tashlich existing as far back as that, but the reason synagogues were built by the sea is for reasons of ritual purity. This is at the heart of the tashlich. The Mekhilta contains several references to prophets recieving prophecies only by bodies of water when the land was impure or they were in exile.
Tashlich was originally a folk tradition, invoking the ire of scholars, because the people considered that they were casting their actual sins into the water.
It is best to avoid the people who are as light-minded as women and say, Ich vil geyn mayn aveyres shiteln [I will shake off my transgressions], and, taking hold of the folds of their clothing, shake them, thinking to themselves that by so doing a mman can shake off the transgressions that he commited all the year before. And he ought not to to think so, for it really is a desecration of the great Name of God before the nations that know of the custom. For when they see Jews going to the river, they say laughingly, The Jews are going shiteln ire zind in vasser [to shake their sins into the water]. But if a person wants to observe the custom, let him say, Ich vil geyn tashlikh makhen [I will perform the Casting]. For the principal purpose of the custom is to pray to God, to cast out our iniquities into the depths of the sea, because in sayings these verses we are contemplating Teshuvah [repentance]... And the custom of shaking the hems of our clothing is symbolic, too; we do it to shake off the shells of the evil spirits that cling to us because of the filth of our iniquities...-Abraham ben Sabbatai Sheftel ha-Levi Horovitz of Prague, Emek Berakhah, Cracow 1597, as included in S. Y. Agnon's Days of Awe.
*Or the sea of the upper realms.