multi-instance openstack neutron charts fail when referencing the same /run hostpath

[wilreichert]

All the neutron openvswitch related daemonsets use a /run hostPath to share openvswitch's conf.db. When running multiple openstack systems on the same kubernetes cluster this fails when the second system hits the .conf.db.~lock~ & dumps something like this into the host logs

Mar 28 16:00:04 oreo05 kernel: traps: ovsdb-server[50808] general protection ip:7efdceb00196 sp:7ffe6f443550 error:0
Mar 28 16:00:04 oreo05 kernel:  in libc-2.23.so[7efdceac9000+1bf000]
Mar 28 16:00:04 oreo05 kernel: traps: handler196[52650] general protection ip:7fb8a1c23196 sp:7fb870ff8ad0 error:0
Mar 28 16:00:04 oreo05 kernel:  in libc-2.23.so[7fb8a1bec000+1bf000]

This seems like a good place to leverage the 'developer' flag & mount a shared volume in a clustered environment.

[wilreichert]

Started digging through the neutron chart some more & realized that with the host networking mode it'll take more than just move an altered /run to have multiple openstack clouds running on the same hardware. Was the intention just to run a single cloud on a kubernetes cluster or is it expected that the operator will apply the appropriate labels such than nodes never overlap? And is 2 network interfaces a requirement or just how the chart was initially tested? Is running without host network mode a possibility? Apologies if these are silly questions.

[intlabs]

@wilreichert Running more than one OpenStack deployment per node is possible (though not with OpenStack-helm as written), but is usually a recipe for a bad time. The agents expect full unfettered access to a hosts resources, and so can often conflict with each other - the simplest and most obvious example being nova-compute, which has no method for accounting for resources potentially consumed by other processes on the host - resulting in poor scheduling of VM's and the potential for either massive over (or under!) consumption of resources. That said it's certainly possible to run multiple neutron-agents on a host, and there may be extreme edge cases where you wish to do this, but I think this would be best achieved this with a chart and configuration written specifically for this purpose if desired, as the host would also need preparation to enable this use-case.

[bluejayA]

@intlabs For production deployment, I totally agree that running more than one OpenStack deployment per node is a bad scenario. However, if you think a use case for CI, running multiple OpenStack deployment per kubernetes cluster does happen. It would be really beneficial if openstack-helm can have a structure to choose to run multiple OpenStack deployments on a kubernetes cluster. Having said that, navigating what would be required, and what would become a common factors vs. specific configurations per each use case might be beneficial for us.

[v1k0d3n]

@bluejayKR i agree with you, especially with regards to CI/CD. we've recently taken an approach of using a container for our CI; meaning that you can spin up a VM and run this container, making multiple OSH deployments possible, and safe to run side-by-side. would working on something like this together in the community be of benefit to you and your team? if so, i can explain in further detail over slack or IRC.