It all depends on your definition of enough. 3" will work, but it will hurt performance a little since it would choke down the 5" outlet of the s475 to 3".
By the way 4" pipe is 36% smaller than 5".
3" pipe is 64% smaller than 5".
You can get a lot of flow through a 3" pipe, it just takes more pressure which means more back pressure on the whole system. But look at it this way, how large is the outlet of the exhaust manifold at the t-3 turbo flange? The exhaust manifold outlet is approximated 3 square inches. A 3" pipe is 7 square inches.
There are only two downsides to smaller pipe,
1. There is a loss in efficiency when the exhaust comes out compact, expands to a 5" exhaust outlet and then gets compressed/funneled back down to a 3" pipe.
2. The smaller pipe requires more pressure to maintain the same flow, therefore there is more friction between the flow of exhaust and the stationary wall of the exhaust pipe. Once again, loss of efficiency.
I'd guess at the 500 hp level, there is less than a 15hp difference between 4" and 5" exhaust. In my opinion, 4" will be just fine.