question

Jean Vence avatar image
Jean Vence asked ·

APL Speech/Text Synchronization

I'm trying to create an ExecuteCommandsDirective to automatically read each card using the Image Forward List Sample found at https://developer.amazon.com/alexa/console/ask/displays (here's the code that is generated https://gist.github.com/jvence/54209ffcd984d123a143ba507c677943)

I'm not sure how to have Alexa read each card automatically and then move to the next one. I tried attaching a transformer but not sure how to point it to the list of items in my data source. Here's what I tried:

RenderDocumentDirective documentDirective = RenderDocumentDirective.builder().withToken("listToken")

.withDocument(document).withDatasources(dataSource).build();


ExecuteCommandsDirective commandsDirective = ExecuteCommandsDirective.builder().withToken("listToken")

.withCommands(Collections.singletonList(AutoPageCommand.builder()

.withComponentId("listComponentId").withDuration(2000).build()))

.build();


I greatly would appreciate any help as I'm on a very tight deadline for a demo.

Thanks

apljava
10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Gaetano@Amazon avatar image
Gaetano@Amazon answered ·

Hello,

thanks for your question.

To let Alexa read out loud elements on the screen, I'd suggest using the SpeakList command. It takes advantage of the Speech property of APL components, but you will need to implement a ssmlToSpeech transformer in order to use it.

I've made an example of a simple APL layout containing a Sequence and 3 Text components filled with the Speech property.

simpleList.json:

{
    "type": "APL",
    "version": "1.0",
    "theme": "dark",
    "import": [],
    "resources": [],
    "styles": {},
    "layouts": {},
    "mainTemplate": {
        "parameters": [
            "payload"
        ],
        "items": [
            {
                "type": "Sequence",
                "id": "mySequence",
                "width": "100vw",
                "height": "100vh",
                "items": [
                    {
                       "type": "Text",
                       "id": "catFactText",
                       "text": "${payload.catFactData.properties.catFact}",
                       "speech": "${payload.catFactData.properties.catFactSpeech}"
                    },
                    {
                       "type": "Text",
                       "id": "catFactText2",
                       "text": "${payload.catFactData.properties.catFact2}",
                       "speech": "${payload.catFactData.properties.catFactSpeech2}"
                    },
                    {
                       "type": "Text",
                       "id": "catFactText3",
                       "text": "${payload.catFactData.properties.catFact3}",
                       "speech": "${payload.catFactData.properties.catFactSpeech3}"
                    }
                ]
            }
        ]
    }
}

Transformers are part of the document datasources{} so they must be implemented there. The simpleListResources.json file contains the value of the 3 Text components that will be shown and read on the screen, along with the 6 transformers: 3 ssmlToSpeech and 3 ssmlToText.

First threes are needed to transform the catFactSsml, catFactSsml2 and catFactSsml3 properties into Speech, while the others are transforming SSML to simple text.

Even though the catFactSsml element is already a string, you still need to use a transformer since SpeakList command supports SSML as well .

For example It could have also been something like:

"catFactSsml" : "<speak>First <emphasis level='strong'>element</emphasis>.</speak>."


Here is the resources file, simpleListResources.json:

{
  "catFactData": {
   "type": "object",
   "properties": {
   "catFactSsml": "First element",
   "catFactSsml2": "Second element",
   "catFactSsml3": "Third element"
   },
   "transformers": [
       {
     "inputPath": "catFactSsml",
     "outputName": "catFactSpeech",
     "transformer": "ssmlToSpeech"
        },
        {
     "inputPath": "catFactSsml2",
     "outputName": "catFactSpeech2",
     "transformer": "ssmlToSpeech"
        },
        {
     "inputPath": "catFactSsml3",
     "outputName": "catFactSpeech3",
     "transformer": "ssmlToSpeech"
        },
        {
     "inputPath": "catFactSsml",
     "outputName": "catFact",
     "transformer": "ssmlToText"
        },
        {
     "inputPath": "catFactSsml2",
     "outputName": "catFact2",
     "transformer": "ssmlToText"
        },
        {
     "inputPath": "catFactSsml3",
     "outputName": "catFact3",
     "transformer": "ssmlToText"
        }
        ]
  }
 }

Please note that the outputName values of ssmlToSpeech transformers are set as Speech property of the Text components, same thing for which ssmlToText are filling the Text property.

How to send directives sent from your Intent handler:

return handlerInput.responseBuilder
            .addDirective({
          type:'Alexa.Presentation.APL.RenderDocument',
          token :'documentToken',
          document: require('./simpleList.json'),
          datasources: require('./simpleListResources.json'),
        })
        .addDirective({
          type : 'Alexa.Presentation.APL.ExecuteCommands',
          token: 'documentToken',
          commands: [
              {
            "type": "SpeakList",
            "componentId": "mySequence",
            "start": 0,
            "count": 3,
            "minimumDwellTime": 1000,
            "align": "center"
}
                    ]
                    
                    })  
      .getResponse();

As you can see the SpeakList command needs a componentID, which is this case is the Sequence's ID from the simpleLayout.json file (mySequence).

So according to your layout, you just need to add an ssmlToSpeech and ssmlToText transformer for every component you want to read out loud, with the following settings:

inputName : Reference of the text you want to read (from your resources)

outputName: You can choose a name, and this must be set as Speech or Text property of the component if using respectively ssmlToSpeech or ssmlToText

Then add an ExecuteCommands directive from your Lambda function to read all the elements.

Lastly, to scroll the sequence you can use the Scroll command.

Kind regards,
Gaetano

10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jean Vence avatar image
Jean Vence answered ·

Thanks. Implemented the code in java and I get the following results. It's show the entire list on my echo device and just speak each item. I need to show a single item per screen, speak the item details, display next item, speak item details, etc... I don't want to see all elements on the same screen. Am I missing something. I also implemented it as follows:

        List<com.amazon.ask.model.interfaces.alexa.presentation.apl.Command> commands=new ArrayList<>();
            commands.add(SpeakListCommand.builder().withStart(0).withCount(3).withComponentId("mySequence").withMinimumDwellTime(1000).withAlign(Align.CENTER).build());
            commands.add(ScrollCommand.builder().withComponentId("mySequence").withDistance(1).build());

Thanks

1 comment
10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi,

if you don't want to see all the elements on the screen, you should modify the APL layout so that the Sequence containing all your elements has 100vw of width and 100vh of height.

Here is an example of hierarchy:

Then you should send a Scroll command after every SpeakList.

Kind regards,
Gaetano

0 Likes 0 ·
Jean Vence avatar image
Jean Vence answered ·

Great thank you. If I want to add a background image, header, etc, do I need to create a separate layout?

I've added the following to the layouts section of simpleList.json (but it had no effect):


        "Sequence": {
            "parameters": [
                "backgroundImage",
                "title",
                "logo",
                "hintText",
                "mySequence"
            ],
            "items": [
                {
                    "when": "${viewport.shape == 'round'}",
                    "type": "Container",
                    "height": "100%",
                    "width": "100%",
                    "items": [
                        {
                            "type": "Sequence",
                            "scrollDirection": "horizontal",
                            "data": "${mySequence}",
                            "height": "100%",
                            "width": "100%",
                            "numbered": true,
                            "item": [
                                {
                                    "type": "FullHorizontalListItem",
                                    "listLength": "666"
                                }
                            ]
                        }
                    ]
                },
                {
                    "type": "Container",
                    "height": "100vh",
                    "width": "100vw",
                    "items": [
                        {
                            "type": "Image",
                            "source": "https://d2o906d8ln7ui1.cloudfront.net/images/md_brie.png",
                            "scale": "best-fill",
                            "width": "100vw",
                            "height": "100vh",
                            "position": "absolute"
                        },
                        {
                            "type": "Sequence",
                            "scrollDirection": "horizontal",
                            "data": "${mySequence}",
                            "height": "70vh",
                            "width": "100%",
                            "numbered": true,
                            "item": [
                                {
                                    "type": "HorizontalListItem"
                                }
                            ]
                        }
                    ]
                }
            ]
        }

Thank you for your invaluable assistance.

10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jean Vence avatar image
Jean Vence answered ·

Any ideas on how to add a background image, header and footer using your suggested approach (with SpeakList). Thanks

3 comments
10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Please check my last answer, there is a JSON representation of what you are asking.

Hope it helps!

0 Likes 0 ·

Thanks but I am trying to add a different image for each item in the list and then use SpeakList to auto scroll, read title and display background image.

0 Likes 0 ·

Using the ListTemplate2 layout as a reference, you should modify the URL of the images from:

data.image.sources[*].url

SpeakList will just read the speech property and scroll the Sequence. To change the background Image at runtime you should use another command like SetValue, to change the Image "source" property.

0 Likes 0 ·
Gaetano@Amazon avatar image
Gaetano@Amazon answered ·

Hi,

to set an Image as background of the whole layout, you should put your Sequence,Image, Header, and footer under a Container with the following hierarchy:

Here is the JSON:

{
    "type": "APL",
    "version": "1.1",
    "theme": "dark",
    "import": [
        {
            "name": "alexa-layouts",
            "version": "1.1.0"
        }
    ],
    "resources": [],
    "styles": {},
    "layouts": {},
    "mainTemplate": {
        "parameters": [
            "payload"
        ],
        "items": [
            {
                "type": "Container",
                "item": [
                    {
                        "type": "Image",
                        "width": "100%",
                        "height": "100%",
                        "source": "https://yourImage.jpg",
                        "scale": "best-fill",
                        "position": "absolute"
                    },
                    {
                        "headerTitle": "Alexa Header",
                        "headerSubtitle": "Header subtitle",
                        "headerAttributionText": "Attribution",
                        "headerAttributionImage": "https://d2o906d8ln7ui1.cloudfront.net/images/cheeseskillicon.png",
                        "headerAttributionPrimacy": true,
                        "headerDivider": true,
                        "headerBackButton": true,
                        "headerBackButtonAccessibilityLabel": "back",
                        "type": "AlexaHeader"
                    },
                    {
                        "type": "Sequence",
                        "id": "mySequence",
                        "width": "100vw",
                        "height": "60vh",
                        "items": [
                            {
                                "type": "Text",
                                "id": "catFactText",
                                "text": "${payload.catFactData.properties.catFact}",
                                "speech": "${payload.catFactData.properties.catFactSpeech}"
                            },
                            {
                                "type": "Text",
                                "id": "catFactText2",
                                "text": "${payload.catFactData.properties.catFact2}",
                                "speech": "${payload.catFactData.properties.catFactSpeech2}"
                            },
                            {
                                "type": "Text",
                                "id": "catFactText3",
                                "text": "${payload.catFactData.properties.catFact3}",
                                "speech": "${payload.catFactData.properties.catFactSpeech3}"
                            }
                        ]
                    },
                    {
                        "hintText": "Alexa Footer",
                        "type": "AlexaFooter"
                    }
                ]
            }
        ]
    }
}

Make sure to import the "alexa-layout" package, and to set the Image to absolute positioning as showed above.

Kind regards,
Gaetano


10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jean Vence avatar image
Jean Vence answered ·

Thanks @Gaetano but I want each item in the list to have its own background image. So in the example above, each CatFact would have a unique image. See https://forums.developer.amazon.com/questions/210796/applying-transformers-to-lists.html for more details on what I'm trying to implement.

1 comment
10 |2000 characters needed characters left characters exceeded

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

The issues seems to be related, please reply to my last question on that post so we can help you further.

Thanks.

0 Likes 0 ·