question

Hi Paul, I work with the speech team at Amazon and would like to try to help! This phrasing is currently still supported for our DATE type, so there may be another issue. Would you mind posting the related training data where you are using this slot? Also, just to clarify, are you seeing just the word "last" when you go to Settings > History in your Echo app, or are you seeing "last tuesday" but then it is not returning the right date? Thanks, Fawn

When I reported this issue I was only seeing the word "last" in the history (no day). I was testing again today and did not see that problem. However, I am seeing problems with spoken dates, such as "june ninth". For example, I tried asking "what was on my schedule on june third" or "june second" or "june fifteenth" and Echo interpreted it as being June 10th in each case. There's also cases where I'll say something like "what's on my schedule june ninth" and Echo will come back with a date like 6/1/2016 (almost a year in the future and not the 9th). Also, I was wondering how "last", "next" and "this" is implement with respect to dates. Is "last" the previous calendar week, "this" the current calendar week (even if the date is in the current week but prior to the current date), and "next the next calendar week? It seems to have problems when using "this" -- sometimes providing a date in the current week, and sometimes in the next week. Here is a section of the sample utterances (there are others, but the slots are the same): GetSchedule {what's|Tense} on my schedule GetSchedule {what's|Tense} on my schedule {may tenth|Date} GetSchedule {what's|Tense} on my schedule {tomorrow|Date} GetSchedule {what's|Tense} on my schedule {monday|Date} GetSchedule {what's|Tense} on my schedule {next tuesday|Date} GetSchedule {what's|Tense} on my schedule for {may tenth|Date} GetSchedule {what's|Tense} on my schedule for {tomorrow|Date} GetSchedule {what's|Tense} on my schedule for {next tuesday|Date}

Thanks for sharing this Paul! I suspect that more training samples with a wider variety of dates will help to solve this problem. With the "june ninth" example, my guess is that only "june" is being slotted correctly, so it is trying to give the next upcoming first day of June. Adding many more dates and phrasings to each of your utterance examples should help to improve the speech understanding. (And yes, there was an issue with things like "last tuesday" on our end that has been fixed recently, so hopefully this should now work more smoothly!) Best, Fawn

Thanks for the reply. I did write a program that generated sample utterances with more phrasings but I ran into a problem with a 200,000 character limit on the Amazon side. How many variations are optimal for a DATE slot? Or for that matter, how many variations are optimal for any slot? For example, if I have a LITERAL slot and can recognize 15 different words for that slot, how many samples utterances should I create? It is very important for dates to be handled properly for my app so any advice that you can provide to increase the accuracy would be appreciated. By the way, I was still running into problems with "last", "next" and "this" today, so if the fix you mentioned has been pushed to production, then there still seems to be a problem. Thanks.

In general, the more data you have for literal slots, the better your app will do. If there are only 15 things that you need to have recognized, using all of those in that slot for each utterance variation will be helpful. (So say you have 10 different ways to phrase a request, having 15 different examples for each of those 10 with the 15 different slot values - 150 total for that request type - will give you your best results.) With things like dates, there is the issue of running out space, but giving at least 100 different examples of dates (including things like "yesterday" and "last tuesday") should help. As for the fix I mentioned before, it seems like it may only be working for phrasings like "on last tuesday" and not just "last tuesday". We are still working on this on our end, but adding all examples here (e.g. for each day of the week following "last") will help train the models for better slotting. -Fawn

Is there any practical limit to the number of sample utterances? I'm looking at a "grocery list" app and there is, of course, the potential for thousands of possibilities. When you get to intents with two slots, you get to N squared possibilities. It's easy to generate a list that big. But at what point does it become a burden on the system than letting it free-form associate with it's English language dictionary?

jjaquinta, The way that the models currently work is that they first heavily prefer any input given to them by the training data you provide, not the whole lexicon of words that is available in the overall training data. In most cases, this results in much better accuracy, as it limits the potential recognitions to things that would be relevant for the apps that you are building. There are, of course, the cases where truly free form data would be better for the app, and these are the cases where the more data, the better for recognition. While there is no specific amount of data that is "enough" or "not enough", the goal is to be as diverse as possible. So for example, if you had two slots in one intent, and let's say 10 different phrases available for that intent, what you would want to do is make sure that you are not pairing the same words for those two slots over and over again, but making different pairs for each phrasing, so that the models don't get too comfortable with one pair. Please let me know if I would be helpful to go into more detail here! Best, Fawn

Well, yes, some more detailed direction would be useful.You might check in on the other thread about utterance file generators. You don't need much complexity to end up with quite a lot of utterances. I'm doing some pre-work for the upcoming hack-a-thon in Boston. As a warm-up for a more general interactive fiction demo I'm wrapping Colossal Cave behind an echo interface. It really has a pretty restricted vocabulary: 10 directions, 9 verbs, 4 compound verbs and about 25 objects. But once you start putting all definitive combinations together I end up with 4500 utterances. That's about 180 utterances per intent. (Not at all evenly distributed.) Getting beyond a demo app, when you start having thousands of values per slot, and, say, three slots, it gets quite huge. Another point this example shows is that it's a lot easier to just generate a massive quantity of utterances, rather than craft by hand a shorter, more representative list. (Must write feature request to get feedback on failures to drive improvement in utterance lists..) So, is it still better to be exhaustive even if the list gets into the thousands? Tens of thousands? Colossal Cave example: Intents file: http://pastebin.com/6QxEK3Zd Utterances file: http://pastebin.com/0v5KYB2Z

So, short answer - yes, it is good to have thousands of samples if they are relevant to your app. It is of course best to make sure that if you use a grammar generator that it is including all of the relevant examples that you noted could be added be hand. In general, the idea is that you want to include all reasonable utterances that people might say to your app, even if the numbers get very high. Because there is currently a limit on file size, if you do end up with tens of thousands of examples, you should focus first on the most relevant utterances (so the most common wordings, the most common potential slot values), but if you can fit in everything you can think of, the more the better!