Yeah, it's time. Every character who is infinite because "moved in a place without time" will be downgraded
The reason for this is because timeless voids, as the name applies, means that time does not exist in those places, making the speed unplacable. Which in turn, makes the characters don't scale to them because there's no time, so they are not moving.
Examples of Characters who will be affected by This:
Giorno (Still Infinite because he is ranked none in speed, but the reasoning because of Diavolo's time skip needs to be removed)