Bulletin of the
Korean Mathematical Society BKMS

Tabanov investigated the {\it global symmetric} perturbation of the integrable billiard mapping in the ellipse \cite{3}. He showed the nonintegrability of the Birkhoff billiard in the perturbed domain by proving that the principal separatrices splitting angle is not zero. In this paper, using the {\it exact separatrix map} of an one-degree-of-freedom Hamiltonian system with time periodic perturbation, we show the existence the stochastic layer including the uniformly hyperbolic invariant set which implies the nonintegrability near the separatrices of a Birkhoff's billiard in the domain bounded by a $C^2$ convex simple curve constructed by the {\it generic global} perturbation of the ellipse.

Keywords: irkhoff's billiard, twist map, Hamiltonian system, separatrix, separatrix map, uniform hyperbolicity

MSC numbers: 34C35, 58F39