Historic circumstances prompted a few western European nations, chiefly Portugal, Spain, England (later the UK), France, the Netherlands and then late comers like Belgium to be at different times in control of vast territories in the Americas, Africa, Asia and what used to be called Oceania. That is a historic fact which has changed the world as we know it now and, for better or worse, we have to deal with it.
European colonialism will always be controversial. Personally to me, as someone who was born and raised in the Americas, what happened in the so-called "settler colonies" (what is now the US, Canada, Australia, New Zealand and, somewhat differently, the major countries in Latin America) was even graver because we are talking here of even mass (direct or indirect) extermination of the native population and entire civilizations (their culture, languages, religion, institutions, etc.) that literally disappeared to be replaced by "new societies" whose mainstream is basically a western European offspring. On top of that, and as a separate (different) issue, the Americas and the Caribbean in particular were also the driving engines of the transatlantic slave trade
Curiously though, the "late" European colonialism in Asia and Africa, of which the Belgian Congo is an example, seems to attract far more attention.