De novo protein design aims to explore uncharted areas of the global protein structure and sequence spaces. Despite recent advances, the success of de novo design remains limited. One of the main challenges is defining the set of “designable” structural protein backbone templates which could in turn help to solve the sequence sampling stages. Many protein backbone design approaches suffer from inaccuracies in both energy functions and sampling algorithms, which often leads to a convergence in sequence space with a few similar sequence variants that frequently fail experimentally. To address these limitations, we build on recent advancements in protein structure prediction and design using deep neural networks. We train a convolutional variational autoencoder (VAE) called Genesis that is trained to improve protein secondary structure lattice models termed Sketches by denoising their 2D feature maps. Genesis interfaces with the recent neural sequence design framework trRosetta to jointly optimize the protein sequence and structure in a higher dimensional feature space, thereby bypassing the arduous and time-consuming step of crafting designable backbones and fitting sequences in 3D space. We used Genesis-trRosetta to design large pools of diverse sequences for a set of protein folds and found that the framework is capable of sampling native-like features maps for known and novel protein topologies. The Genesis framework enables the exploration of the protein sequence and fold space within minutes and is not bound to specific protein topologies. Essentially, our method addresses the backbone designability problem and could ultimately contribute to the de novo design of proteins with new functions.