You don't need anything but regular PC's these days...
2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap.
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years