Boiling system is a typical open dissipative structure. Focusing on the investigation of boundary adjacent to heating plate surface in boiling system, nonlinear non-equilibrium statistical thermodynamics theory of open systems was originally constructed. The competitive bubble formation was realized by non-equilibrium phase transition caused by the interactions among active sites or bubbles. The “natural selection” among all possible sites was mathematically outlined. It was by the means of “natural selection” that a kind of self-organized and self-similar structure was formed. The present studies theoretically explained the experimental observations from other researchers, and gave more reasonable guidance of enhancing boiling heat transfer, such as by way of installing artificial cavities. Beyond the scope of boiling systems, the present studies not only have far-reaching theoretical implications on explaining time, life, thermodynamic, self-similar, fluctuation, order and chaos etc., but also can find many promising applications from new perspectives, such as wider industrial optimization, the recognitions of pattern and reproduction pattern of by computer etc.