Remove rotation based scaling

In order to simplify some of the geometry logic in BufferStateLayer,
and unify with the rest of the layer in SurfaceFlinger we translate the
concept of source and dest frame into crop, scale and position. This is
currently done on the client side.

But if there is buffer rotation transform, we will generate an
additional scale, to scale the buffer size to the new orientation. This
causes issues with rounded corners because the additional scale
stretches the rounded corner incorrectly. And translating the buffer
rotation into a rotation matrix affects child layers.

This solution only adjusts the buffer size based on the rotation
matrix and the scale is generated based on the rotated buffer.
This cannot be done in the client side because we do not have
the current display orientation to unflip the buffer if the client
sets the transformToDisplayInverse flag.

In the future the plan is to drive the transform hint and the
display orientation down from WM so this calculation can go
back to the client.

Test: atest SurfaceControlTest ASurfaceControlTest libgui_test SurfaceFlinger_test
Test: go/wm-smoke
Bug: 185597146, 186191378

Change-Id: Ia566f952f5efe765382434dbc460b4815165f4f5

1 files changed, 8 insertions, 0 deletions

diff --git a/libs/gui/LayerState.cpp b/libs/gui/LayerState.cpp
index 267db7686a..e65c721ae1 100644
--- a/libs/gui/LayerState.cpp
+++ b/libs/gui/LayerState.cpp
@@ -64,6 +64,8 @@ layer_state_t::layer_state_t()
         fixedTransformHint(ui::Transform::ROT_INVALID),
         frameNumber(0),
         autoRefresh(false),
+        bufferCrop(Rect::INVALID_RECT),
+        destinationFrame(Rect::INVALID_RECT),
         releaseBufferListener(nullptr) {
     matrix.dsdx = matrix.dtdy = 1.0f;
     matrix.dsdy = matrix.dtdx = 0.0f;
@@ -167,6 +169,7 @@ status_t layer_state_t::write(Parcel& output) const
 
     SAFE_PARCEL(output.write, stretchEffect);
     SAFE_PARCEL(output.write, bufferCrop);
+    SAFE_PARCEL(output.write, destinationFrame);
 
     return NO_ERROR;
 }
@@ -296,6 +299,7 @@ status_t layer_state_t::read(const Parcel& input)
 
     SAFE_PARCEL(input.read, stretchEffect);
     SAFE_PARCEL(input.read, bufferCrop);
+    SAFE_PARCEL(input.read, destinationFrame);
 
     return NO_ERROR;
 }
@@ -543,6 +547,10 @@ void layer_state_t::merge(const layer_state_t& other) {
         what |= eBufferCropChanged;
         bufferCrop = other.bufferCrop;
     }
+    if (other.what & eDestinationFrameChanged) {
+        what |= eDestinationFrameChanged;
+        destinationFrame = other.destinationFrame;
+    }
     if ((other.what & what) != other.what) {
         ALOGE("Unmerged SurfaceComposer Transaction properties. LayerState::merge needs updating? "
               "other.what=0x%" PRIu64 " what=0x%" PRIu64,