Good points. I think the most promising areas for end-to-end training of more dynamic networks might be gradient free training methods, see https://arxiv.org/abs/1605.02026

Meanwhile as for not knowing how many objects there are in advance, perhaps similar methods using information bottleneck or even something like slot attention (see https://arxiv.org/abs/2006.15055) might be useful