question

Yesterday we started noticing that the automatic skill validation that is run when submitting was failing on all of our skills. We hadn't made any changes to any skills or changed how we handle requests.

The validation always fails due to the validator getting an invalid response for both CanFulfillIntent and our Custom tasks. After further investigation we've found that if we debug our webhook locally or use ngrok.io to proxy the requests to our deployed webhooks everything works and validation passes.

Specifically we get these errors:

This skill implements the Alexa CanFulfillIntentRequest interface, but the CanFulfillIntentRequest interface call’s response is blank or not in the defined structure.

and

One or more test examples for the custom task resulted in an error response when tested.

Your skill must successfully handle requests to invoke your custom task with your test examples. One or more of the test examples resulted in an error response when tested. Make sure that your skill returns a successful response for all test examples.
Documentation Help:

Looking at our logs it seems we aren't even seeing requests from Alexa for CanfulfillIntent testing or the Task tests in our load balancer though we do see them when debugging locally or using ngrok to proxy.

When viewing the requests that the automated validator sends we've noticed that the requests that fail seem to use an older version of java in the user-agent header

Apache-HttpClient/UNAVAILABLE (Java/1.8.0_282)

as well as are lacking a signature-256 header.

Requests we receive for live skill requests or for the rest of the requests that the validator use java version Java/1.8.0_302 as well as include the signature-256 header.

One other unique thing about the requests that we aren't receiving in our live servers is that they seem to be always coming from the same IP address while the rest of the test requests come from a seemingly random IP.

To us it seems that one of the servers that run the automated validation tests is using an older version of java that is either just breaking when trying to send Http requests to our cloud-hosted servers or is being filtered out by infrastructure somewhere.

Again when we test this ourselves or proxy the requests through a service like ngrok, validation passes. Our servers, when they actually receive a request, handle each successfully.

We'd appreciate a quick response on this, we've opened a support ticket as well.