This may have been the first Silverberg story I read. Almost certainly the first novel, and I think it's the best work of his that I've read (and a clear step above others in the series). I still have the paperback I bought around the time this first came out - despite the fact that I foolishly left it out, and some SOB apparently showed off by ripping it partly in half. I don't know what s/he learned. I learned never to leave books around people who can read, but are too foolish to want to. (Not to worry - I bought a replacement. I use this one as a travel book.)
In any case, the magic worked again. This time, I learned that Silverberg has something of Vance's knack with weird description, but without the cold-bloodedness. In fact, one of the strength's of the novel is Valentine's warm heart. Where Vance travels though bizarre lands tense but unmoved, Valentine pours his heart out to anyone that will listen. Of course, since it's a novel, they all do. It works remarkably well. There are a few throwaway characters that get thrown away, and the non-human bystanders that don't get much screen time. But the humans are the heart of the story, and that's done very well.
The main weakness of the story is that while it sets up a fascinating planet, and explores both its surface and its governance, the story doesn't dig very deeply into the main mystery - the Metamorphs whose planet this is. Silverberg addressed that reasonably well in subsequent books, but as a standalone, Lord Valentine's Castle does leave you wanting a little more.
All in all, though the book is a very enjoyable, literate adventure story about identity, discovery, and the power of a warm and generous heart.
Plus, as I found this time through, there's a nice tip of the hat to Mervyn Peake's Gormenghast in reference to the complex Labyrinth and vast Castle.