Mechanisms underpinning the dysfunctional immune response in SARS-CoV-2 infection are not yet fully understood. In addition, the functional roles of the genetic variants identified by COVID-19 genome-wide association study (GWAS) remain elusive, especially in non-European ancestry. We analyzed single-cell transcriptomes and T and B cell receptors of > 895,000 peripheral blood mononuclear cells from 73 COVID-19 patients and 75 healthy controls of Japanese ancestry with host genetic data. COVID-19 patients showed a low fraction of nonclassical monocytes (ncMono). We report downregulated cell transitions from classical monocytes to ncMono in COVID-19 with reduced CXCL10 expression in ncMono in severe disease. Cell-cell communication analysis inferred decreased cellular interactions involving ncMono in severe COVID-19, suggesting that the dysfunction of ncMono might be closely involved in the immunopathology of COVID-19 severity.  Clonal expansions of B cell receptors were evident in plasmablasts of patients. Putative disease genes identified by the GWAS of severe phenotypes showed cell type-specific expressions in monocytes and dendritic cells. A COVID-19-associated risk variant at the IFNAR2 locus (rs13050728) had COVID-19-specific and monocyte-specific expression quantitative trait loci effects, indicating the enrichment of host genetic risk in innate immune cells. Our multimodal and integrative single-cell analyses highlight biological and host genetic involvement of innate immune cells in COVID-19 severity.