The U.S. Census Bureau data was gathered through multiple sources depending on time and type of data.

Infant mortality rate:

1. US National Vital Statistics System https://i.imgur.com/Ebah18w.png

Population total:

1. 1900-1932 National Intercensal Tables: https://www.census.gov/data/tables/time-series/demo/popest/pre-1980-national.html
2. 1933-2020 U.S. Census Bureau July 1st data, provided by Human Mortality Database: https://www.mortality.org

Race data:

1. 1900-1979 National Intercensal Tables: https://www.census.gov/data/tables/time-series/demo/popest/pre-1980-national.html
2. National Intercensal Datasets: 1980-1990: https://www.census.gov/data/datasets/time-series/demo/popest/1980s-national.html (linearly interpolated Hispanic data for this decade due to source Census Bureau data temporarily using a different definition of Hispanic than the other datasets)
3. 1990-2000 Bridged-Race Population Estimates 1990-2020 - Provided by CDC https://wonder.cdc.gov/bridged-race-population.html
4. National Intercensal Datasets: 2000-2010: https://www.census.gov/data/datasets/time-series/demo/popest/intercensal-2000-2010-national.html
5. National Population by Characteristics: 2010-2019: https://www.census.gov/data/datasets/time-series/demo/popest/2010s-national-detail.html

Notes:

1. The data has been interpolated between census years before 1990.
2. Race is self identified.
3. Hispanic is exclusive of other categories.
4. A 5th-degree polynomial fit has been applied at a cohort level after 1960 as smoothening.
5. It was not possible to answer with multiple races before the 2000 census. Yet I've interpolated the race data linearly from the 1990 census, therefore it shows up slowly over the decade.
6. The infant mortality rate for other categories than white and black where assumed to follow the average. This is not a fully reasonable assumption. But the effect should be way too small to be visible in this chart.