I have a Pi 4 Model B/4GB serving a small office as a router/firewall/gateway with dnsmasq (DHCP+DNS), dnscrypt-proxy, chrony (NTP), and a few other odds and ends. It's fairly busy: up to 20 users at a time, lots of file uploads, three main VLANs (plus a management VLAN). There's minimal internal routing for the time being.
I haven't really stress-tested it yet but it's handling all those service and a 500/35-rated connection without breaking a sweat. Speed test is like 590/40. I was worried about 802.1q performance but it hasn't seemed to be an issue. I'm using sch_fq_codel (no shaper) for now; trying to get sch_cake up and running before I take the training wheels off and start throwing tons of stuff at it.
If you're planning to use the second ethernet connection as a WAN port, the problem you're going to run into is that USB NICs have no support for Byte Queue Limits (BQL) in the Linux kernel, and it doesn't appear likely to be added any time soon. This is going to severely hamper performance, and it renders sch_fq_codel, sch_cake, etc. nearly useless. The built-in NIC is attached directly to the SoC—as you've noted—and it uses the bcmgenet driver, which has BQL.
To work around this, I'm actually passing the WAN connection through a managed switch (UniFi) and VLAN-tagging it before sending it to the Pi. This could be a security nightmare but I think I've got it fairly well isolated between a combination of switch settings and firewall rules. As far as performance goes, the easiest way to think about it is that Internet uploads and downloads have to share capacity on both the transmit and receive sides of the gigabit link, so I have a budget of roughly 300 Mbps on either side for the aforementioned services and other internal traffic.
It's running Ubuntu Server arm64 (unofficial installer) with root on a USB 3.0 SATA SSD. For less than $100 all-in—including storage—it's far exceeded my expectations, and it should be fine until it makes sense to upgrade to something more substantial. It probably wasn't worth the cost savings on hardware once my time is factored in, but it's been a fun project!
