With the introduction of Oracle Engineered Systems the DBA is moved somewhat closer to infrastructure design decisions, and expected to at least have some opinions on the network design requirements for the database.
As such, I’ve given some thought to how one can design a fully redundant production network for web applications with Oracle Database Appliance. I’m sure Oracle have some very clear ideas on how everything is meant to work together, but to me it was not all that clear. For a time, and even after reading all documentation and support notes related to ODA, I was convinced that the bonded interfaces meant setting up LACP or EtherChannel groups on the switches, thus requiring fully redundant distribution or core-level switches.
As it turns out, the ODA is factory configured with active-backup bonds. I’ve tested this to work well without any switch-side LACP/EtherChannel configuration, and each bonded connection may be split across two switches. In my tests, no simulated failure or network reconfiguration caused more than a a few hundred milliseconds worth of network outage.
This means that one can set up an isolated redundant front network for web applications using any layer two switches that are not inherently redundant.
To avoid client connections taking the long way into the company network and back through the other switch (and thus making production dependent on that equipment), one can have a private VLAN that only lives on the two edge switches and on an EtherChannel trunk between them.
As such, only the application servers and the database appliance will exist on that virtual network segment.
I don’t see a way to control which path the connections from the application servers take to the database listeners, so the link between the two switches will have to be redundant, less this link becomes a single point of failure. This rules out using unmanaged switches without support for VLAN and either LACP or STP.
Using Cisco Catalyst 2960-series switches, I believe a combination of EtherChannel and Port Fast would be the better choice for a solid independent connection between the two. I would also use Port Fast on the ports for all the bonded connections to ODA and application servers.
Since the production network is isolated, one would need separate network connections for management, backup and connectivity to the rest of the company network.
Naturally, in order for this front production network to be fully self contained, any dependencies to external resources, such as DNS or authentication services, must also be resolved. Ideally production would be able to continue independently from any faults, ongoing maintenance or network outages anywhere else in the data center or company network.
