http://revach.net/article.php?id=1017

Section: Halacha   Category: Tshuvos
Rav Moshe Feinstein - Forgetting HaMelech HaKadosh The First Night of Rosh HaShanah

One who forgets to recite HaMelech HaKadosh during the Aseres Yeme'i Teshuvah must repeat the Shmoneh Esrei. The Chayei Adam says in the name of one of the Gedolim in Vilna (R. Abish Pesvoler) that if one forgets to recite HaMelech HaKadosh on the first night of Rosh HaShanah he doesn't have to repeat the Shmoneh Esrei.

Rav Moshe Feinstein concurs with this Halacha and explains that the reason we recite HaMelech HaKadosh, is because during the Aseres Yeme'i Teshuvah the Malchus of Hashem Yisborach is apparent in the world because this is the time that He is judging the world. The judgment of Hashem Yisborach only starts on the morning of the Rosh HaShanah, because the Heavenly Bais Din does not start the Din on Rosh HaShanah until the Bais Din on earth sanctifies the Chodesh. The sanctification of the Chodesh is only during the day, not at night.

Therefore since the night of Rosh HaShanah is not yet a time for Din, even though L'Chatchilah we must recite HaMelech HaKadosh so as not to make the night of Rosh Hashanah different than the day, however B'Dieved if we fail to recite it we don't have to repeat the Shmoneh Esrei.  Many poskim argue and hold that you must go back so it is best to be extra careful since according to R. Moshe and other poskim going back would constitute a Bracha L'vatala.