We study the stochastic dynamics of the externally regulating gene circuit as an example of an eve-skipped gene stripe in the development of Drosophila. Three gene regulation regimes are considered: an adiabatic phase when the switching rate of the gene from the OFF to ON state is faster than the rate of mRNA degradation; a nonadiabatic phase when the switching rate from the OFF to ON state is slower than that of the mRNA degradation; and a bursting phase when the gene switching is fast and transcription is very fast, while the ON state probability is very low. We found that the rate of thermodynamic cost quantified by the entropy production rate can suppress the fluctuations of the gene circuit. A higher (lower) rate of thermodynamic cost leads to reduced (increased) fluctuations in the number of gene products in the adiabatic (nonadiabatic) regime. We also found that higher thermodynamic cost is often required to sustain the emergence of more gene states and, therefore, more heterogeneity coming from genetic mutations or epigenetics. We also study the stability of the gene state using the mean first passage time from one state to another. We found the monotonic decrease in time, i.e., in the stability of the state, in the transition from the nonadiabatic to adiabatic regimes. Therefore, as the higher rate of thermodynamic cost suppresses the fluctuations, higher stability requires higher thermodynamics cost to maintain.