SARS-CoV-2 infection elicits a robust B cell response, resulting in the generation of long-lived plasma cells and memory B cells. Here, we aimed to determine the effect of COVID-19 severity on the memory B cell response and characterize changes in the memory B cell compartment between recovery and five months post-symptom onset. Using high-parameter spectral flow cytometry, we analyzed the phenotype of memory B cells with reactivity against the SARS-CoV-2 spike protein or the spike receptor binding domain (RBD) in recovered individuals who had been hospitalized with non-severe (n = 8) or severe (n = 5) COVID-19. One month after symptom onset, a substantial proportion of spike-specific IgG+ B cells showed an activated phenotype. In individuals who experienced non-severe disease, spike-specific IgG+ B cells showed increased expression of markers associated with durable B cell memory, including T-bet and FcRL5, as compared to individuals who experienced severe disease. While the frequency of T-bet+ spike-specific IgG+ B cells differed between the two groups, these cells predominantly showed an activated switched memory B cell phenotype in both groups. Five months post-symptom onset, the majority of spike-specific memory B cells had a resting phenotype and the percentage of spike-specific T-bet+ IgG+ memory B cells decreased to baseline levels. Collectively, our results highlight subtle differences in the B cells response after non-severe and severe COVID-19 and suggest that the memory B cell response elicited during non-severe COVID-19 may be of higher quality than the response after severe disease.