Limit recursive polymorphic inlining to prevent code bloat

If both NoRecursion and Wrapper implement the same method
MyMethod, NoRecursion with just `return false` and Wrapper
with `return BaseClass.MyMethod()` we would end up with
generated code similar to:

```
if (receiver == NoRecursion) {
  return false;
} else if (receiver == Wrapper) {
  // A polymorphic invoke which turns into:
  if (unwrappedValue.receiver == NoRecursion) {
    return false;
  } else if (unwrappedValue.receiver == Wrapper) {
    // A polymorphic invoke which turns into
    // [...] you get the gist, we do this several times.
  } else {
    // HInvokeVirtual
  }
} else {
  // HInvokeVirtual
}
```

After the change we would have:
```
if (receiver == NoRecursion) {
  return false;
} else {
  // HInvokeVirtual
}
```

This pattern was worse when there were more than one recursive
polymorphic case, increasing exponentially in size. This was
common to see on Wrapper classes, but uncommon otherwise.

In the two wrappers plus one non-wrapper case in test/2238-,
we will now generate a .cfg which is about 6% the size versus
before this CL (12M -> 779K).

Test: art/test/testrunner/testrunner.py --host --64 --optimizing -b
Bug: 217464625
Change-Id: I3c6444193ea2f24af29d97549776a283ef177efd

1 files changed, 18 insertions, 2 deletions

diff --git a/compiler/optimizing/inliner.cc b/compiler/optimizing/inliner.cc
index ac71ce9b8a..0157d654e2 100644
--- a/compiler/optimizing/inliner.cc
+++ b/compiler/optimizing/inliner.cc
@@ -64,6 +64,11 @@ static constexpr size_t kMaximumNumberOfCumulatedDexRegisters = 32;
 // much inlining compared to code locality.
 static constexpr size_t kMaximumNumberOfRecursiveCalls = 4;
 
+// Limit recursive polymorphic call inlining to prevent code bloat, since it can quickly get out of
+// hand in the presence of multiple Wrapper classes. We set this to 0 to disallow polymorphic
+// recursive calls at all.
+static constexpr size_t kMaximumNumberOfPolymorphicRecursiveCalls = 0;
+
 // Controls the use of inline caches in AOT mode.
 static constexpr bool kUseAOTInlineCaches = true;
 
@@ -952,8 +957,19 @@ bool HInliner::TryInlinePolymorphicCall(
 
     dex::TypeIndex class_index = FindClassIndexIn(handle.Get(), caller_compilation_unit_);
     HInstruction* return_replacement = nullptr;
-    LOG_NOTE() << "Try inline polymorphic call to " << method->PrettyMethod();
-    if (!class_index.IsValid() ||
+
+    const bool too_many_polymorphic_recursive_calls =
+        CountRecursiveCallsOf(method) > kMaximumNumberOfPolymorphicRecursiveCalls;
+    if (too_many_polymorphic_recursive_calls) {
+      LOG_FAIL(stats_, MethodCompilationStat::kNotInlinedPolymorphicRecursiveBudget)
+          << "Method " << method->PrettyMethod()
+          << " is not inlined because it has reached its polymorphic recursive call budget.";
+    } else if (class_index.IsValid()) {
+      LOG_NOTE() << "Try inline polymorphic call to " << method->PrettyMethod();
+    }
+
+    if (too_many_polymorphic_recursive_calls ||
+        !class_index.IsValid() ||
         !TryBuildAndInline(invoke_instruction,
                            method,
                            ReferenceTypeInfo::Create(handle, /* is_exact= */ true),