All User Study Dialog Transcripts

⬅️ Back to MICoBot homepage

These are dialog transcriptions between the robot and human from the speech-to-speech interactions of ALL our user studies. We conducted 18 user studies with our robotic system in the real world, each with a unique participant. Participants were told to be as indifferent or as proactively helpful as they wished.

Table of Contents

Task 1 User Studies: 1A, 1B, 1C, 1D, 1E, 1F

Task 2 User Studies: 2A, 2B, 2C, 2D, 2E, 2F

Task 3 User Studies: 3A, 3B, 3C, 3D, 3E, 3F

Task 1: Open and Pour Package into Bowl

User Study 1A

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Human
Good job.
Robot
Could you please open the package using the scissors on the coffee table? Thank you so much!
helpreq
Human
There are no scissors on the coffee table.
reject
Robot
Let's collaborate to open the package! I can place the scissors onto the coffee table, and you can then use them to open the package. Thank you for your help!
split proposal
Human
I think you should bring the scissors and then you should use the scissors to open the package.
reject
Robot
I'm happy to place the scissors on the coffee table for you! However, I'm not able to open the package with the scissors and place everything on the table. Let me know if there's anything else I can do to assist!
Human
Okay, if you bring the scissors, then I will open the package with the scissors.
accept
Robot
Great! I'll happily place the scissors on the coffee table. Thank you for taking care of opening the package with the scissors!
Robot
[Brings scissors.]
Human
Thanks for bringing the scissors!
Robot
Could you please open the package with scissors and place it on the coffee table? Thank you!
helpreq(x2)
Human
Ok, I will do that now.
accept
Human
[Opens package with scissors.]
Robot
[Pours package into bowl.]
Human
That was a close one.

User Study 1B

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Human
Let me bring the scissor and also correct the position of the cup by myself.
offer
Robot
Thank you for bringing the scissors to the coffee table! However, I'm unable to open the package with the scissors and place both on the coffee table. Hello! Would you mind opening the package with the scissors and placing both on the coffee table? Thank you so much!
helpreq
Human
Okay, let me bring the scissor and open the package.
accept
Human
[Brings scissors and opens the package.]
Robot
Thank you for completing the task! I'm all set for the next step of pouring the package into the bowl.
Human
Yes, please go ahead.
Robot
[Pours package into bowl]
Human
Nice, thank you!

User Study 1C

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Robot
Could you kindly open the package with the scissors and place it on the coffee table? Thank you so much!
helpreq
Human
Yes, I can open the package, but can you please bring the scissors to me? Thanks.
accept, split proposal
Robot
Absolutely, I am more than happy to bring the scissors to you. Thank you for being ready to open the package and place it on the coffee table. Let's get this done!
accept
Robot
[Brings scissors.]
Robot
Could you please open the package with scissors and then place it on the coffee table? Thank you so much!
helpreq
Human
I will open the package with the scissor but can you bring the item to the coffee table please?
accept, helpreq
Robot
Could you kindly open the package using the scissors on the coffee table? Thank you so much!
ignore
Human
[Opens the package with scissors.]
Robot
[Pours package into bowl.]

User Study 1D

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Robot
Could you please help by opening the package with the scissors? Thank you so much!
helpreq
Human
I don't have this either.
reject
Robot
[Brings scissors.]
Robot
Could you please open the package with scissors and place it onto the coffee table? Thank you!
helpreq(x2)
Human
Yes.
accept
Human
[Opens the package with scissors.]
Robot
[Pours package into bowl.]

User Study 1E

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Robot
Would you be so kind as to open the package? Thank you!
helpreq
Human
Can you get the scissors?
helpreq
Robot
Of course! I'm on it. I'll get the scissors for you and place them on the coffee table. Once that's done, you can use them to open the package. Let's do this together!
accept, split proposal
Robot
[Brings scissors.]
Human
Can you open the package?
helpreq
Robot
I'd love to help, but I'm afraid I'm not able to open the package with the scissors and place it on the coffee table.
reject
Human
Sure, I can open it.
accept
Human
[Opens the package with scissors.]
Robot
[Pours package into bowl, but package contents do not land into bowl. Primitive failure.]

User Study 1F

Dialog Annotation
Robot
Human
Robot
[Brings bowl, then brings package.]
Human
Looks like the cup is falling down. Do you want to pick it up?
helpreq
Robot
Could you kindly open the package using the scissors on the coffee table? Thank you!
ignore, helpreq
Human
Can you bring me the scissors first?
helpreq
Robot
Of course, I'd be delighted to bring you the scissors to the coffee table! Once I've done that, you can use them to open the package. Thank you for working with me!
accept, split proposal
Robot
[Fetches scissors, but unable to place them due to primitive error.]

Task 2: Assemble Toy car

User Study 2A

Dialog Annotation
Robot
Human
Robot
[Brings parts tray.]
Human
Let me get the drill.
offer
Human
[Brings drill.]
Robot
[Brings wheels.]
Human
Let me assemble the tires.
offer
Human
[Assembles wheels onto car with drill.]
Robot
Great! Thanks for taking care of assembling the tires by putting the drill on the coffee table and attaching the wheels to the car. I appreciate your help! Could you kindly assemble the wheels by placing the drill on the coffee table and attaching the wheels to the car? Thank you so much!
helpreq
Human
Yes.
accept
Robot
Could you please switch to the correct drill bit? Thank you for your help!
helpreq
Human
Sure.
accept
Human
[Gets phillips drill bit from parts tray. Switches bit on drill from hex to phillips.]
Robot
Could you please assemble the rest of the car? Thank you so much for your help!
helpreq
Human
Of course!
accept
Human
[Assembles window and seats onto car with drill.]

User Study 2B

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Human
Thank you!
Robot
Could you kindly help by assembling the wheels for the toy car? Thank you so much!
helpreq
Human
Can you please bring me the drill?
helpreq
Robot
Of course! I can eagerly bring the drill to the coffee table for you. Let's get those wheels assembled together with a happy spirit! How about we team up on this step? I'll happily bring the drill to the coffee table, and then you can put the wheels onto the car. Let's get this done with a smile!
accept, split proposal
Human
Sounds good!
accept
Robot
[Brings drill.]
Robot
Could you please put the wheels on the car? Thank you!
helpreq
Human
Will do.
accept
Human
[Assembles wheels onto car with drill.]
Robot
Could you kindly switch to the correct drill bit? Thank you very much!
helpreq
Human
No.
reject
Robot
Let's work together on this task. I'll place the correct drill bit on the coffee table, and then could you kindly switch it into the drill? Thank you very much!
split proposal
Human
Let's do that.
accept
Robot
[Moves to pick the phillips drill bit from the parts tray, but isn't successful at grasping it from the clutter.]
Robot
Would you be so kind as to switch to the correct drill bit for us? Thank you!
helpreq
Human
[Gets phillips drill bit from parts tray. Switches bit on drill from hex to phillips.]
accept
Robot
Could you please assemble the rest of the car? Thank you!
helpreq
Human
Ok.
accept
Human
[Assembles window and seats onto car with drill.]

User Study 2C

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Human
Can you also bring the drill?
helpreq
Robot
Absolutely, I'd be happy to bring the drill over to the coffee table!
accept
Robot
[Brings drill.]
Robot
Could you kindly assemble the wheels onto the car? I would really appreciate your help with this step!
helpreq
Human
Okay, I can do that.
accept
Human
[Assembles wheels onto car with drill.]
Robot
Could you kindly switch the drill bit to the correct one? I would be most grateful for your assistance!
helpreq
Human
Ok, I can help with that.
accept
Human
[Gets phillips drill bit from parts tray. Switches bit on drill from hex to phillips.]
Robot
Could you kindly assemble the rest of the car? Your assistance would be greatly appreciated!
helpreq
Human
[Assembles window and seats onto car with drill.]
accept

User Study 2D

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Robot
Could you kindly assemble the wheels onto the car? Thank you so much!
helpreq
Human
I can't, can you get the drill? Then I can do the second step.
reject, split proposal
Robot
I'm excited to get the drill on the coffee table, and thank you so much for handling the wheels!
accept
Robot
[Brings the drill, but while placing it, it falls off the coffee table.]
Human
Can you pick up the drill that fell on the floor?
helpreq
Robot
Unfortunately, I can't put the wheels onto the car. Is there anything else you'd like me to assist with?
ignore
Human
Can you pick up the drill that fell next to the coffee table and is on the floor?
helpreq
Robot
I'm really sorry, but I'm unable to put the wheels onto the car.
ignore
Human
I can put on the wheels on the car.
accept
Human
[Picks up the drill from the floor, and assembles wheels onto car with drill.]
Robot
Could you kindly switch to the correct drill bit for me? That would be amazing, thank you!
helpreq
Human
No, you should switch the right drill bit.
reject
Robot
How about we divide this task up? I can pick up the phillips bit and put it on the coffee table, and then you could switch it onto the drill. That would be fantastic!
split proposal
Human
No, you pick up the new drill bit, put it on the coffee table, and then switch the drill bit.
reject
Robot
I'm happy to pick up the new drill bit and place it on the coffee table, but I'm not able to switch the drill bit onto the drill. Let's keep going strong!
Human
Ok.
Robot
Could you kindly switch to the correct drill bit? Thank you so much!
helpreq(x2)
Human
Can you pick up the correct drill bit and put it on the coffee table? And then I can switch it.
split proposal
Robot
Of course, I can handle picking up the Phillips bit and putting it on the coffee table. Feel free to let me know if you want me to do anything after that!
accept
Human
Sounds good.
Robot
[Attempts to pick up phillips drill bit from parts tray, but misses and grasps some other screws instead.]
Human
You picked up the screws and not the correct drill bit. Can you try picking up the correct drill bit again?
helpreq
Robot
Of course! I'll happily pick up the correct drill bit and place it on the coffee table.
accept
Robot
[Gets phillips drill bit from parts tray.]
Human
Okay, thank you for putting the drill bit on the table. Can you put the drill bit near the drill so I can change it?
helpreq
Robot
I'm happy to help with bringing the drill bit closer, but I can't change the drill bit myself.
accept
Human
You can just move the drill bit, and then I will change the drill bit.
Robot
Thanks for offering to change the drill bit. However, I'm unable to switch the phillips bit onto the drill myself.
Human
Ok, I will do it.
Human
[Switches bit on drill from hex to phillips.]
Robot
Could you kindly assemble the rest of the car, please? I would really appreciate it!
helpreq
Human
No, can you help me turn the car over?
reject, helpreq
Robot
I'm really sorry, but I'm unable to put on the window or seats onto the car. Is there anything else I can assist with? Thank you for your patience!
ignore
Human
Can you pick up the seats and put it on the table?
helpreq
Robot
I don't have the capability of performing put on - window - car. I know you didn't want to do it, but can you please do it?
ignore, helpreq
Human
As an initial step, can you pick up the seats from the box and put it on the coffee table?
reject, helpreq
Robot
[Terminates the episode, as human has rejected performing the same step twice in a row.]
ignore

User Study 2E

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Robot
Would you be able to help by assembling the wheels onto the car, please? It would be much appreciated!
helpreq
Human
No.
reject
Robot
I'm happy to propose a split for this step! I can grab the drill and place it on the coffee table while you kindly attach the wheels to the car. How does that sound?
split proposal
Human
Sounds good.
accept
Robot
[Brings drill.]
Human
[Assembles wheels onto car with drill.]
Robot
Could you please put the wheels on the car? Thank you!
helpreq
Human
I already put the wheels on the car.
accept
Robot
Could you please switch to the correct drill bit? I really appreciate your help!
helpreq
Human
Ok.
accept
Human
[Gets phillips drill bit from parts tray. Switches bit on drill from hex to phillips.]
Robot
Could you kindly help with the next step and assemble the rest of the car, please? I really appreciate your cooperation!
helpreq
Human
No, you should do something.
reject
Robot
Could you please put the window on the car? Thank you!
helpreq
Human
Ok.
accept
Human
[Assembles window onto car with drill.]
Robot
Could you please put the seats in the car? Thank you!
helpreq
Human
No, can you do it for me?
reject, helpreq
Robot
[Terminated episode, as it mistakenly thought the human had rejected the same step twice in a row.]
ignore

User Study 2F

Dialog Annotation
Robot
Human
Robot
[Brings parts tray, then brings wheels.]
Robot
Could you please assemble the wheels for the car? Thank you so much!
helpreq
Human
No, could you please do that?
reject
Robot
I'm thrilled to help with picking up the drill and placing it on the coffee table. However, I'm afraid I can't put the wheels on the car. Thank you for your understanding! How about we divide this task? I'll be delighted to place the drill on the coffee table, and you can put the wheels on the car. Does that work for you?
split proposal
Human
Sure that sounds great!
accept
Robot
[Brings drill.]
Robot
Could you please put the wheels on the car? Thank you!
helpreq
Human
Sure, I can do that.
accept
Human
[Assembles wheels onto car with drill.]
Robot
Could you please switch to the correct drill bit? Thank you so much!
helpreq
Human
Yeah, I can do that.
accept
Human
[Gets phillips drill bit from parts tray. Switches bit on drill from hex to phillips.]
Robot
Could you kindly assemble the rest of the car? That would be a great help, thank you!
helpreq
Human
Sure, I could do that.
accept
Human
[Assembles window and seats onto car with drill.]

Task 3: Pack Gift Box

User Study 3A

Dialog Annotation
Robot
Human
Robot
[Folds down box flap. Then puts tissue paper and car into box.]
Robot
Could you kindly help with sealing the box by covering it with the lid, and wrapping the ribbons around it? I'd really appreciate your cooperation. Thank you so much!
helpreq
Human
Yes, but get me the ribbons first.
accept; split proposal
Robot
Thank you for handling the step of cutting a piece of tape and putting it on the box. However, I'm not able to wrap the ribbons around the box. Thank you for your understanding!
ignore
Human
I'll close the box while you grab the ribbons from the other table and bring them back to me.
split proposal
Robot
Great, thank you for handling the task of closing the box with the lid! I'll eagerly bring the ribbons back to you from the other table. Let's keep up the fantastic teamwork!
accept
Human
Thank you.
Human
[Closes box lid.]
Robot
[Brings ribbons.]
Human
Thank you.
Robot
Could you please help me wrap the ribbons around the box? Thank you!
helpreq
Human
Yes.
accept
Human
[Wraps ribbons around box.]
Robot
Would you be so kind as to help me decorate the box? Thank you so much!
helpreq
Human
No, you got it.
reject
Robot
How about we split this task? You could cut a piece of tape and put it on the box, while I place the gift bow on it. Thank you for your help!
split proposal
Human
Sounds good.
accept
Human
[Cuts a piece of tape, then puts it on the box.]
Robot
[Places gift bow on box lid at tape location.]
Human
You might want to move it a little bit to your left. Right, right, right, right.
helpreq

User Study 3B

Dialog Annotation
Robot
Human
Robot
[Folds down box flap.]
Human
Alright, so put the car over there inside of the box.
helpreq
Robot
[Puts tissue paper into box.]
Human
So the paper you have to fix it by pushing it fully into the box and laying it flat, not standing up as it was before.
helpreq
Robot
[Puts car into box.]
ignore, accept
Human
Now what you have to do is take the lid next to the box and flip it around and put it on top and line it up with the edges.
helpreq
Robot
Could you kindly help with sealing the box? Please cover the box with the lid, place the ribbons on top, and wrap them around it. Thank you so much!
reject, helpreq
Human
I actually think you should do that because I'm busy I'm scrolling through Twitter, so you should do the lid for the box.
reject
Robot
How about we tackle this task together? You could cover the box with the lid, I'll place the ribbons on the box, and then you can wrap the ribbons around it. Thank you so much for your cooperation and understanding!
split proposal
Human
[Closes box lid.]
accept
Robot
[Brings ribbons.]
Human
All right, I will do the ribbons. I'm going to do one, and then I'm going to ask you to do the other one.
split proposal
Human
[Wraps first ribbon around box.]
Robot
Sounds great! Thank you for taking care of placing the ribbons on the box. Unfortunately, I'm not able to wrap the ribbons around it afterward, but I'll be ready to assist with anything else you might propose!
reject
Human
Ok, I will put the other ribbon down. No worries man, I understand.
accept
Human
[Wraps second ribbon around box.]
Robot
Could you please help me wrap the ribbons around the box? Thank you!
helpreq
Human
Okay. I wrapped around, I can fix, I guess this one's a little off. I'll fix it, but they're wrapped around brother, what you're talking about.
accept
Human
[Adjusts the straightness of the ribbon around box.]
Robot
Would you be able to help me out by decorating the box? Thank you so much!
helpreq
Human
Sure, I'll put the tape on top of the box, and then you should put the ribbon on top once I put the tape, if you feel so inclined. Please help me pull the tape a bit by grabbing onto the loose exposed end.
split proposal
Robot
Thank you so much for putting the tape on the box! I'll be happy to put the gift bow on top once you do that.
ignore, offer
Human
I actually haven't done it yet, but I will now.
offer
Human
[Cuts a piece of tape, then puts it on the box.]
Robot
[Places gift bow on box lid at tape location.]
Human
That looks great. I'm just gonna press it down a little bit more.
offer

User Study 3C

Dialog Annotation
Robot
Human
Robot
[Folds down box flap.]
Human
Well, I'm too busy. I hope you put the tissue and car in the box.
helpreq
Robot
No worries! I'll happily put both the tissue paper and the car into the box for you.
accept
Human
Thank you, go ahead.
Robot
[Puts tissue paper into box.]
Human
Thank you, please put the car as well.
helpreq
Robot
[Puts car into box.]
accept
Human
Good job! Now we need to seal the box. Can you please bring the ribbon on the table to coffee table?
helpreq
Robot
Absolutely! I'll bring the ribbons and place them on the box with enthusiasm. Thank you for guiding me with such precision!
accept
Human
No problem. When you do that I will close the box and seal the box.
offer
Robot
Thank you so much for taking care of covering the box with the lid and wrapping the ribbons around it. I truly appreciate it!
Human
No worries.
Robot
Could you please cover the box with the lid? Thank you!
helpreq
Human
Yes of course I'll cover the box.
accept
Human
[Closes box lid.]
Robot
[Brings ribbons.]
Human
Thank you, I'll seal the box.
Robot
Thank you for sealing the box! Once you're ready, you can wrap the ribbons around it, cut a piece of tape and put it on the box, and place the gift bow on top. I appreciate your help!
helpreq
Human
Thank you. I'll seal the box, but I hope you to bring the bow for me.
accept, split proposal
Human
[Wraps ribbons around box.]
Robot
Absolutely! I'll happily bring the gift bow and place it onto the box for you. Thank you for sealing the box!
accept
Human
No worries, go ahead.
Robot
Could you please cut a piece of tape and put it on the box? Thank you!
helpreq
Human
Of course I can do that.
accept
Human
[Cuts a piece of tape, then puts it on the box.]
Robot
[Places gift bow on box lid, but it falls off the box lid. Primitive failure.]
Human
Do you need my help?
offer

User Study 3D

Dialog Annotation
Robot
Human
Robot
[Folds down box flap.]
Human
Hi.
Robot
[Puts tissue paper and car into box.]
Human
Hey, do you want help with anything?
offer
Robot
I appreciate your offer! Since you're asking, it would be wonderful if you could help cover the box with the lid. Thank you so much! Thank you so much for offering your help! Could you please assist by covering the box with the lid? I'd really appreciate it!
helpreq
Human
Okay, I can do that.
accept
Human
[Closes box lid.]
Robot
[Brings ribbons.]
Human
Hello.
Robot
Could you please wrap the ribbons around the box? Thank you!
helpreq
Human
I can -- can't you do it yourself though?
reject
Robot
I don't have the capability of performing wrap - ribbons - box. I know you didn't want to do it, but can you please do it?
helpreq(x2)
Human
Alright, I'll do it.
accept
Human
[Wraps ribbons around box.]
Robot
Could you please decorate the box by cutting a piece of tape and putting it on the box, and then placing the gift bow on it? Thank you so much!
helpreq
Human
I can put the tape on but you need to put the bow on it.
split proposal
Human
[Cuts a piece of tape, then puts it on the box.]
Robot
Thank you for handling putting the tape on the box! I'll happily take care of placing the gift bow on it now. Your cooperation is greatly appreciated!
accept
Human
You're welcome.
Robot
[Places gift bow on box lid at tape location.]
Human
Cool, do you need any help with anything else or do you think the job is done?
offer

User Study 3E

Dialog Annotation
Robot
Human
Robot
[Folds down box flap, then puts tissue paper into box.]
Human
It would be nice if the parchment was laying flatter.
helpreq
Robot
[Puts car into box.]
Human
The parchment is kind of squished underneath the car. If you can hold the car for me, I can fix the parchment quickly.
split proposal
Robot
I'm sorry, but I can't cover the box with the lid. Could you help with that part? Thank you so much! Could you kindly help with sealing the box? Specifically, could you cover the box with the lid? Thank you so much for your assistance!
ignore, helpreq
Human
Yeah, I can, but can we fix the parchment first? If you can pick up the toy car, I can fix the parchment.
helpreq, split proposal
Robot
I'm delighted to hear you'll help! Unfortunately, I won't be able to wrap the ribbons around the box.
ignore
Human
Okay, that's fine. We can get to that later. Let me put the lid to the box on and then we can talk about the next steps.
accept
Human
[Closes box lid.]
Robot
That's wonderful! Thank you for putting the lid on the box. Once you're ready, feel free to wrap the ribbons around the box. I'm here to help with anything else you might need!
helpreq
Human
Can you grab the ribbons for me so I can wrap the box?
split proposal
Robot
Absolutely! I'll grab the ribbons for you so you can wrap them around the box. Thank you for asking!
accept
Robot
[Brings ribbons.]
Human
Okay, let me wrap that for you.
Human
[Wraps ribbons around box.]
Robot
Could you please help me wrap the ribbons around the box? Thank you!
helpreq
Human
I will wrap it for you.
accept
Robot
I don't have the capability of performing cut - put - tape - box. I know you didn't want to do it, but can you please do it?
helpreq
Human
Yeah, I can. Can you just put the ribbon on top of the box so that it is easier to reach?
accept, helpreq
Robot
Thank you for taking care of the task! I'll make sure to place the ribbon on top of the box as requested earlier.
accept
Robot
[Places gift bow on box lid.]
Human
Thank you. I'll attach the ribbon to the box now.
Human
[Cuts and places tape under gift bow.]

User Study 3F

Dialog Annotation
Robot
Human
Robot
[Folds down box flap. Then puts tissue paper and car into box.]
Robot
Could you please help us by sealing the box?
helpreq
Human
No, I would rather you seal the box.
reject
Robot
I can happily place the ribbons on the box, but I'm unable to cover it with the lid or wrap the ribbons around the box. How about we work together on this step? If you could please cover the box with the lid and then wrap the ribbons around it, I can happily place the ribbons on the box. Thank you so much for your help!
split proposal
Human
No, I would like you to try if at all possible. I'd like to see that you put in the effort before I intervene. Thank you.
reject
Robot
I don't have the capability of performing cover - lid - box. I know you didn't want to do it, but can you please do it?
helpreq
Human
No, I would like you to try. At least, at least grab the ribbons for me.
reject, split proposal
Robot
[Terminated episode because human rejected the help request on the same step twice in a row.]