Articles Blog

Can Your Phone Translate Signs On The Fly?

Can Your Phone Translate Signs On The Fly?

NAT: Hey, there. I’m Nat. LO: And I’m Lo, and this
is our 20% Project where we go around Google
learning about all the stuff we’re curious about. NAT: And in this episode
we’re finding out how the Google Translate app
does the crazy sci-fi magic that it does. LO: Specifically,
there’s this feature where you press the camera
button, put your phone up to something you
can’t read, and voila. It instantly translates it
into a language you can. NAT: This feature
actually started out as an app called Word
Lens, and then last year the team that made
it came to Google and are now part of
Google Translate. LO: So we met up
with Otavio, who helped invent Word Lens
to find out how it works. NAT: Otavio is a self-professed
non-language guy. OTAVIO: I used to
speak Portuguese, but I kind of forgot it. NAT: In his spare time, he
likes to animate fractals. LO: He goes snowboarding,
and at his wedding, he got a quadcopter
to deliver the rings. NAT: Needless to
say, we were pretty excited to go talk with him. LO: So, OK. So how did it all start? OTAVIO: So I was, you know,
being a tourist in Germany. I was, at some point,
in a bookstore, and I was looking around me
and I could read anything. And there were some books
that looked really cool. They looked like, maybe,
physics books or something. I love physics. And so I wanted to be
able to point my camera phone at these things
and translate them. And so I put
together a prototype over the course
of a couple weeks, and that was enough to
convince me to quit my job and start working
on it full-time. NAT: And then two
and a half years later, with the help of some
other talented programmers– LO: And Otavio’s
super awesome mom. NAT: –who speaks,
like, 10 languages, they turned it
from this prototype to a real working app. LO: Which is now a part
of Google Translate. OTAVIO: So here’s a sign. It’s in Russian. This is the kind of
thing that you will see in a Russian subway station. NAT: I’m going to go with
dangerous piranhas are in this area. OTAVIO: So there’s the sign. NAT: Should I guess it? I have no idea, actually. LO: Diving. NAT: Diving is not or something. LO: So what’s it say? OTAVIO: It said, Attention. Prohibition of bathing. NAT: Oh, bathing. LO: Yeah. OTAVIO: And I like this
one because it does show, like, you know, the
translation isn’t perfect, but it gets the idea across. LO: So can you explain,
like, how it actually works? OTAVIO: So the way it works is
the phone gets an image back from the camera. We just throw out the color. There usually isn’t a whole lot
of information in the color. And first we have to
process it a little bit. You know, we have to get rid
of soft shadows and things that aren’t really
contributing to the text. So we put it through
a bunch of filters. And then at that point,
we can kind of just threshold the image
to black and white. So we have dark things
and light things. And then we kind of look
for little blobs that are sitting next to each other. And those blobs are
probably letters. They might be letters,
they might be trees, they might be hubcaps. So the next step is to
classify what these things are. So when we do the classifier,
the letter classifier, it comes back with a score. And, you know, sometimes
that score says, I am very confident
that this is a letter P. And sometimes the score
comes back and says, I have no idea what
this is, really. And so from that confidence we
can start throwing things out. And then from
there you basically have a bunch of letters,
and you can look that up in your dictionary, do
a simple translation. And then we already
read all the letters, so we can pick up
what the background color is around these letters. Just draw a big box around
the original letters that’s in the background color, and
then draw the foreground font. LO: You make it sound
easy, and yet it, like, happens instantaneously. OTAVIO: I want to make a
window into a world that’s in your language. I want to be able
to go like this, and everything that you
look at is in your language. What helped me was in
Russia I could buy things at the supermarket and
understand what they are, you know? I mean, you see all these
things, they all look the same. And one of them is milk,
and one of them is cream, and one of them is some
other strange kind of milk. You know? And you have to figure
out which one it is. NAT: And now Otavio and
the rest of the team are trying to get new
languages working, which is a lot harder than
I think we’d ever realized. OTAVIO: Latin is one of the
easier scripts for the computer to read. You know, in Latin, there’s
only about 80 to 100 characters that you really care about. Chinese is harder, because there
are thousands of characters. Then there are things like
Arabic, where the whole word is kind of– it’s like cursive
where the word is continuous and it doesn’t
have clear breaks. And that’s really hard for
the computer to figure out. So whenever somebody
suggests I do Arabic, I want to run away
because of the difficulty. NAT: I don’t know
why, in my head I was thinking hieroglyphics. That would be a hard problem. OTAVIO: So when I was
at a museum a while ago, I did a lot of pictures
of hieroglyphics– NAT: Yeah? OTAVIO: –just in case
I someday had the urge to make a hieroglyphics
translator. But it’s actually really hard. LO: So although Google Translate
can’t translate hieroglyphics– NAT: Just yet. LO: –it did just get 20
new languages working, including Swedish, Romanian, and
a few others I can’t remember. NAT: We wanted to say
thank you for, you know, taking the time out
to come talk with us, so we made you this
official thank you. OTAVIO: Uh-oh. I wonder what it says. “Thanks for your help
with our project. This official certificate thanks
from Natalie and Lorraine. They baked for you cookies.” All right. I like cookies. LO: Yeah. I like cookies, too. Can I have a cookie? [MUSIC PLAYING] -Cookie. Oh. Oh.

99 thoughts on “Can Your Phone Translate Signs On The Fly?”

  1. Hahaa in this video I've noticed а funny and wrong translation (…1.43min/6.33min. ..) "вьiход в город".. this is mean "exit to city" 🙂 .." вьiход" = exit … corrрect one is "вход в город"… and then translation like "accesses to city" will be a correct expression!! Don't believe Googles translator toо much, yet. Bring with you for your journey and some paper version … Enjoy!!

  2. This one is also good with cool UI including camera translation – Global Translator – Have a look 😀

  3. now THATS what you call a APP, this is what a mobile app is supposed to be! well done brilliant! even if you charged me $50 i wouldnt mind. YOU GUYS ARE GENIUS

  4. Hey!loved it. can you do a video of all the cool things that you can create with machine learning/things that are being developed with help of machine learning?. Plzzz.

  5. Seems like a genuine guy, here is an awesome suggestion. What if you are unsure of the language you are looking at to begin with, could the camera auto detect???

  6. Both of you are beautiful & happy souls, I love your videos, please keep uploading them , more often, just subscribed .

  7. No and Lo, your videos are just awesome. I hope they will inspire kids to want to learn science to make the world more awesome.

  8. Great video ladies! I'm always on the plane and this helped me with communicating with others!
    Do you need wifi all the time or does it use your data? Once again amazing video

  9. What a bunch of awesomeness! You guys have a really creative way of, well, creating things… and "we made for you cookies. Yes, I love cookies… Can I have a cookie?"😂

  10. well, I totally understand the difficulties to translate Chinese to English by vision of the APP, but I really wish for a single-direction translation from English to Chinese with world lens. It will help a lot and easier than the double-direction function.

  11. Was the massive google books scanning action / Streetview story somehow contributive to machine learning for googles "word lens"?

  12. Both of you are awesome!
    Great work!
    This was my first video that I watched and instantly sub`ed you!
    Btw Nat you`re cute!

  13. I found out about you because of Google Apps New Voice. Now I'm hooked. These videos are fantastic! Keep them coming. 😀

  14. Yeaaa science won't stop it will go ahead for more awesome inspirational things that don't come to your mind yet,… that awesome girls i love you and keep it up it will be an accomplishment (:

  15. I know, technically, to recognize Chinese characters are hard job. But why no English to Chinese translate?

  16. Accounts Help
    Problems signing in to your account
    If you can't sign in, please select the issue that applies to you below. This troubleshooter will guide you through 2-5 questions, depending on the options you select.
    Please select your issue:

  17. even though I already knew what translate was and how to use it and everything but this vid makes everything a lot better

  18. Right now, it seems google translate supports only Latin-based languages, which is quite awesome. Next progress is Chinese Korean Japanese support?

  19. Interested in participating in a Google User Research study? You don't need to be a tech nerd or even use Google products to help out. Best of all – selected participants will receive a super cool thank you gift. Sign up here:

  20. Well, they wouldn't really be able to "translate" hieroglyphics… because the translator is for phonetic languages, not images that represent things.

  21. Der Anfang ist falsch geschrieben. 😛 Richtig wäre es so: "Willkommen zu unserer Show"
    The Begin is wrong. 😛 correct would be: "Willkommen zu unserer Show"

  22. I remember the anecdote. The interpreter signs rushing out to his aunt. He left a card with such content on a not ready sign. Another person has recognized that it is already made translation
    and manufactured hundreds of signs with the inscription "rush to my aunt, I finish tomorrow"
    I made that post with google helps

  23. tried the app. impressive as it is, it doesn't do anything shown in this video.
    it does translate. but…. in a seperate dialogue box. kinda clumsy. not with the ease shown in this video. kind of a letdown.

  24. I never new it could do this! How amazing is this technology!! Just tried it myself and this will come in so handy when im away from home and give you a sense of control back in your life when you are away as not being able to read stuff is quite scary

  25. problem is google translate is an unreliable translator. I speak french and I translated a document to english with google translate and it was hilarious how badly translated it was

  26. Wow god bless you I salute you Google translate team u r reading signboards in all languages with mobile phones

  27. Drawing inspiration from Google Translate, imagine what we can do if we add some smart camera features into Google Maps… You raise your phone and arrows point out directions on the real streets; you turn around and museums info, restaurants reviews, buildings histories just pop up on your map. They are all coming soon, arent they?

Leave a Reply

Your email address will not be published. Required fields are marked *