Week 17: Speech voice recognition, app audio, responsive UI, SMB ASG2

Week 17 is the week which we have to hand in our final project. I worked on the features I was assigned to work on, which is the speech voice recognition first.

I tried looking for a few Unity plug-ins tutorials that I can use to develop this feature in our application.

The first unity plug-in that I found is used in this tutorial

However, once I tested it, I realised it did not do what it was supposed to do when I tested it on both Unity game window and my android device. So, I went to the comments section of this video to see if anything is relevant to why it didn't work in my project.

There is Vuforia used in our Unity project so I definitely can't remove my Vuforia just for this feature to work. I head out again to google to find another plug-in.

I found another tutorial that uses Watson IBM Speech-to-Text feature.

However, when I imported it in Unity, there will always be an error that did not allow me to edit the script. I tried to edit the script myself to remove the error present but it still prevails which I have no idea how to solve. I couldn't get to the step which they mention to input the JSON code even though I have already created an account and obtained my JSON code.

I continued looking for tutorial that will be able to support my speech recognition feature.

I looked at this but headed out as soon as I saw it was not for free.

I could not risk it as they was no one to verify that it does work with Vuforia and all my other packages that Shiyu have already imported before.

I went back to the very first one that I presented for the CPD presentations that is UnityAndroidSpeechRecognition by gsssrao.

I did test this out using the APK uploaded and also imported the package into my Unity project. I tested out the sample scene by exporting it as an APK into my android device and it did not work like it is supposed to. Both APK allow me to press the button, record my voice, recognises it as speech but did not change the text user interface to what it recognised into text from my voice. In the end, I just proceeded with not doing validation for the voice to match the line in the storybook and allow the narration and subtitle animation to proceed regardless of what the user says.

When testing, there was a period of time when I tested and the google speech-to-text shows an error. It was only 3 hours later that I realised that the feature doesn't work if you are screen recording with audio. I assume it is because that it has to record audio so it can't work if another application (my screen recording application) is already recording audio.

I added audio into our application to make it more immersive. There will be a click sound if you click on any of our UI buttons. There is also background environmental sounds when you are in the augmented reality (AR) camera view which becomes louder as you move closer toward the environmental plane. There is also narration when you pressed on the speaker icon. There is an audio that plays along with the appearance of our splash screen and there's a background music when you are on the UI screens.

I made our UI in unity responsive by editing the canvas to be screen space - overlay when it is initially based on screen space - camera.

Our user interface looks like this on phone devices. To demonstrate, I used Samsung Galaxy S8 screen preview with Unity's preview package.

On tablet devices, I showcase it on Apple iPad (7th gen).

Here's the presentation slides for our final app.

We also did Social Media & Branding (SMB) assignment 2 this week which I updated our PR Kit and Branding guidelines in.

I also updated our personas along with Shiyu who designed it. I completed our proposal and included my teammate's work into the proposal itself. I was in-charge of doing the Tiktok contents and the website.

Below are the contents of what we did for SMB assignment 2