I'll note that I didn't say I read only for world building, but I do read first for world building much of the time. That's what I love about Tolkien. I read the Thomas Covenant books for world, though I wouldn't read them again, because I hated the characters. The Ringworld books I read and reread primarily for world.
Nigel Tranter's historicals have cookie cutter characters and plots, but I read them for the deep research on historical Scotland. It's Rome that draws me back to Colleen McCullough's First Man in Rome series again and again, though the characters are good too. I won't read Frankowski ever again because his writing is misogynistic, and his plots are meh, but I read the Crosstime Engineer series more than once because the non-misogynistic parts of the world building is fantastic.
Andre Norton's work I remember mostly in terms of world, though a few of her characters have stuck with me. I could go on if I went down and actually looked at my shelves instead of just going off the top of my head, but that's seems a fair sample. And that's without talking about short stories, where I'm even more likely as a reader to let story and character slide if the world is cool enough.
Oh, and honestly, I think the weakest part of the Harry Potter books (which I do love) is the world building. There's a lot of wow, but very little internal consistency, and tons of contradictions that blow giant holes in her plots. I read those mostly for character—as I mentioned in my original comment I don't read only for world.