question

j-liang avatar image
j-liang asked

question about the end of A/B test

If the A/B test end, we will convert the test to a launch, and all user get the finally selected variation. Is it meaning only "new customers" receive the final variation? What about users in the experiment? If the users in the experiment can also receive the final variation, How? app clients should query the variation by themselves? or they can by pushed final variation by the A/B test service? In our test, we find that when we change a "test" to a "launch", the experiment client can not get stable variation. It can still get the variation version from A/B test after it firstly got the the launch version value.
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Sujoy@Amazon avatar image
Sujoy@Amazon answered
Understood your query. I am flowing up with concerned team aand will get back shortly. Thanks.
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Sujoy@Amazon avatar image
Sujoy@Amazon answered
Thanks for your question. When an AB Test is "launched", ALL customers should get that variation. For example if testing A and B and then launching B, users who had A will transition to B, users who have B will continue to have B, and new users will get B. When launching the variation on the website, a client will only get the new variation the next time it's requested in the app. (the service does not push this change automatically). Did you immediately request the variation after changing it in the website? Both the SDK and the service supplying the variations do some caching. It can take up to 5 minutes before the service supplying the variations will actually serve the launched variation. So if you launched Variation B, and then immediately requested a variation in the SDK, it's possible that you might get A (since the service still thinks the AB Test is not launched and allocates the user into A). Then a few minutes later you request again and the service now gives you Variation B because it sees that it's launched (and ALL customers should get B). Try waiting at least 5 minutes before requesting a variation and see if you still get the condition where you are "flip-flopping" variations. No matter what variation you had, after 5 minutes you are guaranteed to get the launched Variation. Other than during this transition time of an AB Test to Launch, once a user receives a Variation, they keep that variation until the service specifies otherwise. If you are still seeing this behavior where the user changes variation, let us know and we can help you troubleshoot this some more. If you have any more questions, please let us know.
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

j-liang avatar image
j-liang answered
Thank you for your help
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.