[eventlet] Ensure unit tests are monkey patched

This change refactors how watcher manages monkey_patching
modules to achieve 2 goals.

First, we want to ensure the watcher code is tested as it is used
in production. While many tests can run without eventlet,
the existing unit tests depend on eventlet monkey patching
indirectly by importing watcher code that uses eventlet.spawn and
greenthread executors. While that mostly functions today it has
incorrect and inconsistent behaviour on Python 3.9 vs Python 3.12.

Second, the unit tests that test the cmd module were indirectly
monkey patching the test executor during the execution of the tests
as a side effect of importing watcher.cmd. As such the order the tests
execute in and how they are distributed across test workers changed
if the test was monkey-patched or not.

This change makes all tests run with monkey_patching by adding
monkey patching in the watcher/tests/__init__.py
This change also splits the monkey patching from the import
in preparation for an eventual removal of eventlet in a future
release.

Change-Id: I967f3469bd66e69c00863d553bc859343afbb3ff

watcher/cmd/__init__.py

@@ -15,6 +15,15 @@
 # common/service.py.  This allows the API service to run without monkey
 # patching under Apache (which uses its own concurrency model). Mixing
 # concurrency models can cause undefined behavior and potentially API timeouts.
-import eventlet
-
-eventlet.monkey_patch()
+# NOTE(sean-k-mooney) while ^ is true, since that was written asyncio was added
+# to the code base in addition to apscheduler which provides native threads.
+# As such we have a lot of technical debt to fix with regards to watchers
+# concurrency model as we are mixing up to 3 models the same process.
+# apscheduler does not technically support eventlet but it has mostly worked
+# until now, apscheduler is used to provide a job schedulers which mixes
+# monkey patched and non monkey patched code in the same process.
+# That is problematic and can lead to errors on python 3.12+.
+# The maas support added asyncio to the codebase which is unsafe to mix
+# with eventlets by default.
+from watcher import eventlet
+eventlet.patch()

watcher/eventlet.py

@@ -0,0 +1,41 @@
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+import os
+
+
+MONKEY_PATCHED = False
+
+
+def is_patched():
+    return MONKEY_PATCHED
+
+
+def _monkey_patch():
+    if is_patched():
+        return
+    # Anything imported here will not be monkey patched. It is
+    # important to take care not to import anything here which requires monkey
+    # patching. eventlet processes environment variables at import-time.
+    # as such any eventlet configuration should happen here if needed.
+    import eventlet
+    eventlet.monkey_patch()
+
+
+def patch():
+    # This is only for debugging, this should not be used in production.
+    if (os.environ.get('OS_WATCHER_DISABLE_EVENTLET_PATCHING', '').lower()
+            not in ('1', 'true', 'yes', 'y')):
+        _monkey_patch()
+        global MONKEY_PATCHED
+        MONKEY_PATCHED = True

watcher/tests/__init__.py

@@ -13,8 +13,15 @@
 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 #    License for the specific language governing permissions and limitations
 #    under the License.
+# NOTE(sean-k-mooney): watcher does not split up the tests that need eventlet
+# and those that do not currently so we need to monkey patch all the tests.
+# as an example the watcher.test.cmd module is importing watcher.cmd,
+# that has the side effect of monkey patching the test executor
+# after many modules are already imported.
+from watcher import eventlet
+eventlet.patch()
 
-from watcher import objects
+from watcher import objects  # noqa E402
 
 # NOTE(comstud): Make sure we have all of the objects loaded. We do this
 # at module import time, because we may be using mock decorators in our