We seek to understand the neural circuit basis for memory stability in the songbird. This talk describes technical challenges to recording neural activity in singing birds, and engineering efforts to provide miniature devices for brain and nerve recording. We apply these tools to ask how the brain encodes a stable behavior – the stereotyped song of a zebra finch. We find a mesoscopic dynamical pattern that relates excitatory and inhibitory neurons, state dependent control of variability in neural firing patterns, and an unexpected turnover in the neural representation of a stable behavior that occurs during intervals of sleep. These observations can be loosely related to aspects of regularization in modern machine learning that reduce over-fitting of artificial neural networks.