question

tifr0z avatar image
tifr0z asked

Anyone else seeing substantial instability from the Fling iOS SDK?

This is a screenshot from my Crashlytics dashboard. As you can see the stability of the app is sub-par to say the least (~10% of all users, are affected by at least one crash on any given day or approx ~3% of all sessions). I am only showing the to top 2 issues on the screenshot, but easily 90% of all crashes can be attributed to the Fling SDK version 1.3.1

I don't have any reasons to believe that the app itself is (indirectly) causing this crashes of the Fling SDK, but I am not sure. In particular my app is using other SDKs for a similar purposes (Chromecast etc...), and I suspect some unwanted interactions between the different frameworks being a factor possibly.

Anyway, if you are using the iOS Fling SDK version 1.3.1, I would be really curious to know your crash rate numbers in % of users and % of sessions.

fire tvappamazon fling
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Levon@Amazon avatar image
Levon@Amazon answered

Hi tifr0z,

Hopefully other users will chime in on this too. We would like to add an explanation of the issue that is causing these crashes. We are aware that there is a bug in one of our underlying libraries that can cause crashes while an application is shutting down. This is caused by the library using global variables in a multi-threading environment. The global variables are released when the main thread is done shutting down even though the background threads may still be using them. There are multiple variables that can affect how often this causes a crash: how long it takes for your application to shut down after it calls suspend on the DiscoveryController, the amount of memory the device is currently using, the amount of active network calls your application has running during shutdown, the amount of activity on the network during shutdown. These crashes are all background crashes that are invisible to the user since it happens after they have killed the application. That being said we are still working to resolve these crashes as no one wants to see them on their crash dashboard. We would also be happy to hear if this is affecting other applications using the Fling SDK.

On a side note tifr0z, on another thread talking about this same issue you mentioned that you had some crashes that were not 100% background crashes. We would be happy to look at those individually since those should not fall into the bucket described above. For those if you could post a crash log on this thread or the other thread we could start analyzing what is happening. Thanks!

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tifr0z avatar image
tifr0z answered

Thanks Levon

For context, most sessions that crash have a very limited interaction with the fire TV. I capture a decent amount of application activity logs for each crash, so I have a pretty good idea what the usage pattern is like. In particular, for most sessions that end with a crash, the app will typically

  • call discoveryController.start() on didFinishLaunchingWithOptions
  • many sessions will call discoveryController.stop() at some point during the session
  • all session will call discoveryController.stop() on applicationDidEnterBackground
  • Crash
  • ...and that's it as far as the Fling SDK is concerned

This doesn't really fit with your description? In particular, if discoveryController.stop() is being called mid-session, the session should not end with a crash if I follow your explanation.

I am attaching 2 stack traces, captured with SDK v1.3.1. Issue 7397 has a foreground occurence rate of 12%, issue 4774 has a foreground occurence rate of 16% - unfortunately I can't tell if the 2 stack traces I am sharing took place with the app running in the foreground or in the background because crashlytics is silent about this (though this could probably be inferred by looking at the other thread's activities).

archive.zip


archive.zip (20.9 KiB)
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Levon@Amazon avatar image
Levon@Amazon answered

Hi tifr0z,

Thank you for the stack traces, they are very helpful. We believe we have a fix for the stack trace that is crashing in BonjourHelper. That will be in our next release. We will investigate further into the crash with the SSL issue.

Update: Upon further investigation of the SSL issue it looks like it may be a packaging issue. Are you linking an SSL library in your application other than the SSL library that is being included with the Fling SDK? Thanks!

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tifr0z avatar image
tifr0z answered

Hi Levon

Glad I am checking on this thread occasionally because I don't seem to getting the update notifications! I will try to check back here on a regular basis

Yes, the app does use the openssl. The openssl library is not linked - the source code is simply added to the project, however 2 dependencies for openssl are linked: libcrypto.a and libssl.a. (I would be happy sharing more specific information in a more private forum)

In case it might be relevant, in addition to Fling, the app uses other 3rd-party frameworks (these are largely black boxes to me - they may or may not contain SSL libraries)

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tifr0z avatar image
tifr0z answered

@Levon@Amazon, are there any steps you recommend I take on my end to confirm the issue is caused by conflicting libraries - or to prevent these conflicts?

1 comment
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@tifr0z, I will post here as soon as I hear back from the Fling team. Thanks!

0 Likes 0 ·
Levon@Amazon avatar image
Levon@Amazon answered

Hi tifr0z,

This additional context is helpful. All the OpenSSL functions should be available in the Fling framework to link. Could you please try to remove the references to libcrypto and libssl? Thanks!

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tifr0z avatar image
tifr0z answered

Thanks @Levon@Amazon ...to my considerable surprise the app compiled just fine after unlinking libcrypto + libssl.

Unless we uncover issues during testing we will ship the next version with this configuration (i.e. using the crypto libraries embedded in the Fling SDK) - I will circle back here after we get some numbers from running in production. Keeping my fingers crossed!

1 comment
10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Sounds great, thanks tifr0z!

0 Likes 0 ·
tifr0z avatar image
tifr0z answered

Hi @Levon@Amazon

Unfortunately, early results after recently publishing the last version of the app (after unlinking libcrypto+ and libssl) show no improvements with the crash rates associated with the Fling SDK.

Any suggestions? How can we effectively troubleshoot this?

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Levon@Amazon avatar image
Levon@Amazon answered

Hi tifr0z,

Thanks for the update. We cannot reproduce this issue locally. However, we are working on an update to the Fling SDK release, which potentially addresses the call stack, which has causes crash observed by you. No ETA at the moment, but I will keep this thread updated. Thanks!

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tifr0z avatar image
tifr0z answered

Thanks for the update, I am hoping the next iteration will substantially improve the crash rate!

10 |5000

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.