Update entrypoints after switching the runtime to non-debuggable

When detaching a debugger or stopping tracing in non-debuggable
runtimes, we switch back to non-debuggable mode. We used to switch to
non-debuggable mode after updating the entrypoints of methods. This
meant we don't use AOT code for these methods. For this to work
correctly we should
1. Update instrumentation level
2. Switch runtime if possible
3. Update entry points.

Since this order is important, we no longer expose MaybeSwitchRuntime as
a public member. Instead we pass a parameter that says if we should try
and switch the runtime to non-debuggable.

Bug: 303686344
Test: art/test.py
Change-Id: Id655ca50964ed10072ff5bfc3e565c08ecb9987b

1 files changed, 10 insertions, 4 deletions

diff --git a/openjdkjvmti/deopt_manager.cc b/openjdkjvmti/deopt_manager.cc
index 025bed53b2..42ab5d30dc 100644
--- a/openjdkjvmti/deopt_manager.cc
+++ b/openjdkjvmti/deopt_manager.cc
@@ -388,9 +388,14 @@ void DeoptManager::Shutdown() {
     return;
   }
 
-  runtime->GetInstrumentation()->DisableDeoptimization(kInstrumentationKey);
-  runtime->GetInstrumentation()->DisableDeoptimization(kDeoptManagerInstrumentationKey);
-  runtime->GetInstrumentation()->MaybeSwitchRuntimeDebugState(self);
+  // If we attach a debugger to a non-debuggable runtime, we switch the runtime to debuggable to
+  // provide a consistent (though still best effort) support. Since we are detaching the debugger
+  // now, switch it back to non-debuggable if there are no other debugger / profiling tools are
+  // active.
+  runtime->GetInstrumentation()->DisableDeoptimization(kInstrumentationKey,
+                                                       /*try_switch_to_non_debuggable=*/true);
+  runtime->GetInstrumentation()->DisableDeoptimization(kDeoptManagerInstrumentationKey,
+                                                       /*try_switch_to_non_debuggable=*/true);
 }
 
 void DeoptManager::RemoveDeoptimizeAllMethodsLocked(art::Thread* self) {
@@ -475,7 +480,8 @@ void DeoptManager::RemoveDeoptimizationRequester() {
   deopter_count_--;
   if (deopter_count_ == 0) {
     ScopedDeoptimizationContext sdc(self, this);
-    art::Runtime::Current()->GetInstrumentation()->DisableDeoptimization(kInstrumentationKey);
+    art::Runtime::Current()->GetInstrumentation()->DisableDeoptimization(
+        kInstrumentationKey, /*try_switch_to_non_debuggable=*/false);
     return;
   } else {
     deoptimization_status_lock_.ExclusiveUnlock(self);