All User Study Dialog Transcripts (Baseline)

⬅️ Back to MICoBot homepage

These are dialog transcriptions between the robot and human from the speech-to-speech interactions of ALL our user studies. We conducted 18 user studies with our robotic system in the real world, each with a unique participant. Participants were told to be as indifferent or as proactively helpful as they wished.

Table of Contents

Task 1 User Studies: 1A, 1B, 1C, 1D, 1E, 1F

Task 2 User Studies: 2A, 2B, 2C, 2D, 2E, 2F

Task 3 User Studies: 3A, 3B, 3C, 3D, 3E, 3F

Task 1: Open and Pour Package into Bowl

User Study 1A

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Human
Good job.
Robot
[Brings scissors.]
Human
Cool! You brought the scissors.
Robot
[Terminated episode, because the robot allocated the "open package" task to itself, which it can't do]

User Study 1B

Dialog Annotation
Robot
Human
Robot
[Brings bowl.]
Human
Can you bring the cup?
helpreq
Robot
[Brings package.]
accept
Human
Can you bring me scissors so I can open the package?
helpreq
Robot
Sure! I will bring the scissors to you.
accept
Robot
[Brings scissors.]
Human
Can you hand over the scissors to me?
helpreq
Robot
I'm unable to bring you the sensor right now. But I will continue with the current task.
reject
Human
What are you going to do next?
Robot
Next, I will open the package. After that, I will pour the contents into the bowl!
Human
Ok, please go ahead.
Robot
[Terminated episode, because the robot allocated the "open package" task to itself, which it can't do]

User Study 1C

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package, then brings scissors.]
Robot
Could you please open the package with the scissors and place it on the coffee table? Thank you!
helpreq
Human
[Opens package.]
accept
Robot
[Pours package into bowl, but package contents do not land into bowl. Primitive failure.]

User Study 1D

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package, then brings scissors.]
Robot
[Terminated episode, because the robot allocated the "open package" task to itself, which it can't do]

User Study 1E

Dialog Annotation
Robot
Human
Human
Can you bring me the scissors before the package?
helpreq
Robot
Ok, I will bring the scissors to you first after I move the package and bowl to the coffee table.
reject
Human
Sure.
Robot
[Brings bowl, then brings package. However, package was tipped over while placing.]
Robot
Could you please put the scissors on the coffee table? Thank you!
helpreq
Human
Can you bring it for me?
reject, helpreq
Robot
[Brings scissors, but is unable to place it properly since package was tipped over. Primitive failure.]
accept

User Study 1F

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package, then brings scissors.]
Robot
I'm working on cutting the package with scissors on the coffee table, then pouring it into the bowl!
offer
Human
Ok go ahead.
Robot
[Terminated episode, because the robot allocated the "open package" task to itself, which it can't do]

Task 2: Assemble Toy car

User Study 2A

Dialog Annotation
Robot
Human
Robot
[Brings parts tray.]
Robot
[Brings wheels, but fails due to a perception error.]

User Study 2B

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Human
Thank you!
Robot
[Brings drill, but fails due to perception error.]

User Study 2C

Dialog Annotation
Robot
Human
Robot
Sure, thanks for getting the drill! I will bring the parts tray to the coffee table next.
offer
Robot
[Brings parts tray, then brings wheels.]
Robot
Could you please pick up the drill, and place it on the coffee table? Thanks!
helpreq
Human
Ok, I will get the drill now!
accept
Human
[Brings drill.]
Robot
[Terminated episode, because the robot allocated the "put on wheels" task to itself, which it can't do]

User Study 2D

Dialog Annotation
Robot
Human
Robot
[Brings parts tray.]
Human
Can you get the wheels?
helpreq
Robot
Sure, I can get the wheels!
accept
Robot
I am getting the wheels now!
Human
Thank you.
Robot
[Brings wheels.]
Human
Can you get the drill from the kitchen?
helpreq
Robot
[Brings drill.]
accept
Human
Can you give me the wheel screws from the box?
helpreq
Robot
I don't see the wheel screws in the box, can you please assist with locating them?
reject, helpreq
Human
Do you see wheel caps in the box?
ignore
Robot
I will check for wheel caps in the box, and inform you of their presence!
Human
Ok thank you.
Human
Did you find wheel caps in the box?
Robot
I am still in the process of checking for wheel caps in the box, and will update you shortly!
Human
I will assemble the wheels on the car. Can you give me the new drill bit?
offer, helpreq
Robot
I will switch to the new drill bit for you. Please proceed with assembling the wheels on the car.
accept
helpreq
Human
Ok thank you, can you switch the drill bit? It's not right though...
reject, helpreq
Robot
I will switch the drill bit to the correct one for you. Please proceed with assembling the wheels on the car once it's ready.
accept, helpreq
Human
No, I will switch the wheels first and then you can change the drill bit.
reject
Human
[Puts wheels on car.]
Robot
[Drops the drill bit.]
Human
Can you retry picking up the new drill bit?
helpreq
Robot
I will retry picking up the new drill bit myself.
accept
Human
Thank you.
Human
Can you give me the green car seat from the box?
helpreq
Robot
The green car seat will be handed to you shortly.
accept
Human
Can you pick up the car seat from the box and put it on the coffee table?
helpreq
Robot
I will place the car seat on the coffee table shortly!
accept
Human
Ok thank you!
Human
Can you pick up the car seat and put it on the table?
helpreq
Robot
I will proceed with the current tasks and then place the car seat on the table for you!
accept
Human
What tasks?
Human
Can you pick up the window and put it on the coffee table?
helpreq
Robot
I will take care of picking up the window and placing it on the coffee table shortly!
accept
Human
Thank you.
Human
I need to think about what I want to do next, I don't know.
Robot
[Terminated episode, because the robot allocated the "put on wheels" task to itself, which it can't do]

User Study 2E

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Human
Go get the driver.
helpreq
Robot
[Brings drill but drops it.]
accept
Human
Can you pick up the driver?
helpreq
Robot
I am currently focusing on assembling the toy car, starting with the wheels. If you mean the driver bit for the drill, I will handle that soon.
accept
Human
But you need a driver to attach the wheels.
Robot
I understand your point about needing the driver. I plan to switch to the correct drill bit shortly. Let's get the wheels ready first, and then I'll handle the bit switch.
Human
Ok.
Robot
[Terminated episode, because the robot didn't pick the drill up.]

User Study 2F

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels, then brings drill.]
Robot
[Terminated episode, because the robot allocated the "put on wheels" task to itself, which it can't do]

Task 3: Pack Gift Box

User Study 3A

Dialog Annotation
Robot
Human
Robot
[Folds down box flap. Then puts tissue paper and car into box.]
Robot
[Terminated episode, because the robot allocated the "cover lid" task to itself, which it can't do]

User Study 3B

Dialog Annotation
Robot
Human
Robot
[Folds down box flap.]
Human
So next, I need you to put the yellow paper in the box. laying flat. Please, good sir.
helpreq
Robot
Absolutely, I will put the yellow paper in the box as you requested.
accept
Human
Cool thank you, and then after that you can put the car in the box.
helpreq
Robot
[Puts tissue paper into box.]
Human
So I need you to flatten out the paper by picking it up and then rotating it a little bit and then and then putting it in a box in that rotated position so it lays flat.
helpreq
Robot
I will proceed with placing the car inside the box, covering it with the lid, and then wrapping and decorating it. Please let me know if there's anything specific you'd like to change or assist with.
accept
Human
You do that, I'll fix the paper, but yeah, now you can put the car in.
split proposal
Human
[Flattens the tissue paper.]
Robot
Thank you for your help! Let's start with my tasks, and I'll leave the wrapping and taping to you for the best results.
Robot
[Places car inside box.]
Human
I really want you to try and pick up the lid and put it on the box. It's easy, you just grab it from one of the sides, then turn it 180 degrees, and then put it on the box. And then I can fit it from there.
split proposal
Robot
I will pick up the lid and place it on the box as requested.
accept
Human
Thanks dawg, appreciate it.
Robot
[Terminated episode, because the robot allocated the "cover lid" task to itself, which it can't do]

User Study 3C

Dialog Annotation
Robot
Human
Robot
[Folds down box flap.]
Human
Can you please put the tissue and car inside the box?
helpreq
Robot
[Puts tissue paper inside box.]
accept
Human
Thank you, go ahead for car as well.
helpreq
Robot
[Puts car inside box.]
accept
Human
Good job! I'll cover the box.
offer
Robot
Could you please cover the box with the lid? Thank you!
helpreq
Human
Of course, I can do that!
accept
Human
[Covers box with lid.]
Robot
[Brings ribbons.]
Human
Good job, I'll seal the box.
offer
Human
Yeah, I'll seal the box.
offer
Human
[Wraps ribbons around box.]
Robot
[Terminated episode, because the robot allocated the "cut tape" task to itself, which it can't do]

User Study 3D

Dialog Annotation
Robot
Human
Human
Hey, do you need help?
offer
Robot
I appreciate the offer! I've got most of this, but can you help with wrapping the ribbons?
helpreq
Human
Sure yeah we can do that after you get to that step.
accept
Robot
I'll handle most of the tasks, and ask for your help when wrapping the ribbons.
Human
Coolio.
Robot
[Folds down box flap, then puts tissue paper inside box.]
Human
Thank God you did that. Do you want me to speed things up?
offer
Robot
Thanks for offering to help! Let's work together to speed things up.
Human
Okay, what do you want me to do?
offer
Robot
Thanks! Could you please cover the lid on the box while I finish the other steps?
helpreq
Human
You sure about that? I think there's something you need to do first.
reject
Robot
I'll continue handling the steps and ask for your help when needed.
Human
No I want to help you with the next step. I'll put the car in the box.
offer
Robot
Could you please pick up the car box and place it down? Thank you so much!
helpreq
Human
Cool, I will pick up the car and put it into the box.
accept
Human
[Covers box with lid.]
Robot
[Brings ribbons.]
Human
Cool, what's up?
Robot
Great! I'll have you wrap the ribbons around the box while I finish up the remaining steps.
helpreq
Human
Sounds good I'm on it.
accept
Human
[Wraps ribbons around box.]
Human
Yes I have done that. You can do the rest, right?
helpreq
Robot
[Terminated episode, because the robot allocated the "cut tape" task to itself, which it can't do]
ignore

User Study 3E

Dialog Annotation
Robot
Human
Robot
[Folds down box flap,then puts tissue paper inside box.]
Human
You should straighten out the parchment paper in the box.
helpreq
Robot
I will straighten out the parchment paper in the box before continuing.
accept
Human
Make sure it's laying flat.
helpreq
Robot
[Tries to place the car inside the box. However, it drops it.]
ignore
Human
I'll grab the cart for you and straighten out the parchment paper.
offer
Human
[Picks up the car, then straightens out the tissue paper.]
Robot
[Places the car in the box.]
Human
It looks good, continue assembling the box.
helpreq
Robot
[Terminated episode, because the robot allocated the "cover lid" task to itself, which it can't do]

User Study 3F

Dialog Annotation
Robot
Human
Robot
[Folds down box flap, then puts tissue paper inside box.]
Human
Let me put the paper in as well as the car.
offer
Robot
Could you please pick up the car box and place it down? Thank you!
helpreq
Human
Yeah.
accept
Human
[Places car inside box.]
Robot
[Terminated episode, because the robot allocated the "cover lid" task to itself, which it can't do]