Thing is, everything you say relates to whether someone should pursue the work --> clerk --> work path. Whether one should seek a delayed start date after having chosen that path is an entirely separate question.
I disagree with this. The big career concern about taking a clerkship break while at a firm is that the firm is going to get the vibe that you don't want to be there, and will just return/stay at the firm if you can't get your first choice job out of the clerkship. (This isn't a concern for straight-out-of-law-school clerkships.) You want to avoid doing anything that will exacerbate this suspicion. Asking to take an extended break (more than a month) after the clerkship before returning to the firm might do that. Again, it might not as well, so it probably makes sense to get the opinion of someone outside of the official chain-of-command. But this isn't a trivial concern.
it seems to me partners/firms actually think very, very little about people who have not started yet.
Yes, but a work-clerk-work person has started. And if/when they come back, they will likely do so as a fourth or fifth or even sixth year associate (the last assumes someone who does a two-year clerkship after three years at a firm). This is an entirely different scenario than a more-or-less fungible entry-level hire. Partnership decisions will be in a few years for the returning clerks, and since firms tend to like weeding out associates before they are up for it, the unofficial preliminary decisions are going to be made in the fifth and sixth and seventh years. You see the problem there -- groups are going to have to figure out whether they want to elevate the associates returning from the clerkships almost as soon as they walk back in the door, and oftentimes even before that. You can see why they might give a little more thought to the associates on clerkship leave than they do the incoming newbie associates. (And this says nothing of the fact that unlike entry-level hires, the work-clerk-work people have spent multiple years working with the groups, and that the groups are going to have to figure out how to replace that person's capacity while out.)