The Deep End of Deep Learning | Hugo Larochelle | TEDxBoston
I grew up in Quebec one of the largest french-speaking province in Canada. And when it was time for me to go to college. I had to the honor of being proposed a scholarship by McGill University often considered by many as one of the most prestigious universities in Canada now. McGill is English speaking and at the time I was concerned that my English wasn't strong enough and that this would affect my grades so I ended up declining the scholarship and instead going to the University the Mafia which is a good institution but most importantly for me then it's also french-speaking now thankfully my English since then is much improved don't worry but then I I just couldn't help but wonder whether I made the horrible mistake. Well it turns out that this was the best decision. I've ever made because it's led me to join one of a handful of labs in the world that were doing research on an AI technology known as artificial neural networks so you might not have heard about artificial neural networks but perhaps you've heard of an AI technology known as deep learning deep learning for instance is behind the technology the voice recognition technology behind many devices such as Siri on the iPhone or the Amazon Alexa and many other voice enabled devices well at the core of deep learning is the use of artificial neural networks so what artificial neural networks they are computer programs that enable the machine to learn and they are inspired by some of the computation that goes on in our brains so in real neural networks now consider the situation of building a machine that can read and writing at the core of artificial neural networks is the artificial neuron you see one here and much like real neurons artificial neurons are connected here we have a neuron that is connected to the pixels in that image and the job and this is much like in fact real neurons that some of our neurons are connected to our retina and the job of another official neuron is to detect patterns from its incoming connections now much like real brains which have many many neurons artificial neural networks have many artificial neurons each doing a different thing detecting a different type of pattern and then finally much like in real brains which are organized in distinct regions each doing a different jobs or performing a different function in deep artificial neural networks we have multiple layers and this is the core idea behind deep artificial neural networks behind deep learning and this is meant to mimic some of what we see in the brains of one since the light that hits our retina will go through multiple different regions of our brain before it eventually reaches an area with neurons that understand more abstract concepts like - for instance a symbol.
Now this might not be surprising that to develop an artificial intelligence that we might need an artificial brain and in fact as far back as the 1980s and even before then there were a lot of researchers performing research on designing better artificial neural networks but by the time that I started my PhD that activity had reduced quite a bit and there was only a handful of labs still performing research on artificial neural networks and the reason is that then there were a lot of other different machine learning methods that seemed more successful at simple AI tasks and in fact the research on artificial neural networks seemed to be mostly successful with simple artificial neural networks with a single layer so kind of like a brain but with just one brain region and in fact there were a lot of researchers that had essentially just given up on the artificial neural networks approach and it wouldn't be uncommon for researchers like me to submit work at conferences and get reviews at red a little bit like this where are we we just rejected just for using artificial neural networks. This isn't an exact quote. I couldn't find it back in my emails. You can probably imagine. I didn't care much for it then just got rid of it and yet now fast forward.
Ten years and deep learning is all the rage in academia. It's one of the most popular topic of research in industry deep. Learning technologies are being acquired at the millions of dollars and in the media and press. It's often reported as the new. AI much like in this piece and Scientific American so what happened well what I thought I'd do today is give you my perspective on the last ten years in deep learning that is from its emergence and how it evolved and progressed through the years. I'll talk not just about the different technology breakthroughs but also focus a bit on how the community itself evolved and progressed so for me. Things really started in 2006. The thing that really influenced my research was this paper by. Geoffrey Hinton who you see here from University of Toronto with Cimino Sendero and UOIT so in that paper Geoffrey Hinton was proposing a new approach to artificial neural networks and what was really exciting about this work is that it achieved deep artificial neural networks. That would rival some of the more standard more popular machine learning methods of the time so this really sparked a new hope in that the approach using artificial neural networks might actually be successful for achieving. AI this was a new hope so it was new so people essentially came up with a new name to refer to that type of research and they called it deep learning so for instance the next year Ike organized with some of my colleagues the first deep learning workshop it was. We tried to organise it that's part of the neural information processing systems conference which is one of the largest machine learning conferences and so we submitted a proposal for the this workshop but it was rejected however. Geoffrey Hinton just wouldn't have it so he put together the resources necessary for us to actually organize it as a parallel event and was a huge success. We attracted about 10 times as many people as other official workshops that happen during the conference so it was clear there was a lot of excitement in academia for the potential of deep artificial neural networks.
And then in the next three years we started seeing an emergence of more more papers on deep artificial neural networks referred instead under the name of deep learning now there were a lot of papers published but the progress was relatively slow. I turns out that executing artificial neural networks on regular computers is slow and so in about 2010 several different labs figured out a way of executing artificial neural networks not on standard computers but on graphics cards on GPUs the same graphics cards that we use to generate crisp graphics for computer games. So this marks for me the first way major way in which the deep learning community has been changing it has become way. Better at exploiting computational resources what this math is that a deep learning research lab could essentially build its own mini supercomputer but at a few thousand dollars and in fact it's that year that Geoffrey Hinton and his lab produced the first result suggesting that deep learning microevolution eyes speech recognition research. This came as a big surprise and in fact the speech research community a kind of difficulty believing some of these results or at least they were harder to publish initially but now deep learning is in a big way present in speech recognition research. And it's also part of the technology like behind Syrian Alexa then. In 2011 we start seeing the emergence of a lot of really good high-quality software's for in libraries for supporting deep learning research like piano and torch and a few others and to me. This marks the second way in which deep learning has really been changing over the years. It now has a new dedication towards creating high-quality robust easy-to-use open and free code libraries to support the learning research so used to be a target or neural networks. Were somewhat difficult to use and implement but now it's actually quite easy to get started by leveraging the work of other people through these open-source libraries so deep learning community has made performing deep learning research much less like carpentry and much more like playing with LEGOs in 2012 Jeff Renton Jeff fintan or prepper prepares the next revolution with deep learning this time in computer vision so him in his lab participate to a computer vision competition.
The challenge here is to design a system that can read a photograph and identify what are the objects and animals in this photograph and so the results come in and it turns out that their system totally crushes the competition and reaches accuracies. That were never seen before now. This time this breakthrough was undoubted. In fact now in computer vision it is also a field that's in large part dominated by deep learning methods right now so in 2013 there starts being a lot of excitement around deep learning methods and that excitement that year is about to transition to industry and in a big way so for instance that year with my colleagues from 2007 we decided to organize another edition of the deep learning workshop. This time our proposal is accepted and in fact not just that but we get folks from Facebook that reach out and say that their CEO Mark Zuckerberg himself actually wants to be present and participate. So let me try to convey how unusual this is organizing an academic workshop and have mark zuckerberg show up is kind of like organizing a party with your personal friends and then well look at that. Mark Zuckerberg is here. This is totally a surprise. And and not just that for someone like me who used to do research initially in my PhD and I could barely get my colleagues and other topics on machine. Learning interested in artificial neural networks is almost beyond comprehension. In fact the end the interest from industry is as high as ever and also at that workshop we see the first demonstration by little-known startup deepmind technologies of first version of their system that is able to play.
Atari games at the level of humans and in fact less than a year later the plan was acquired by Google. Also that year in 2013 the International Conference on learning representations is created I had the honour of co-chairing that conference in the past two years and I mentioned it for two reasons. The first is that this conference is now mostly known as the deep learning conference and so that means that in 2013 the community is big enough a vibrant enough than it can sustain. Its own conference. The other reason most important reason is that this conference has a very unique reviewing model for scientific work authors are asked to submit their work publicly right away on a website known as archive.org so now the work is accessible for everyone and then the whole deep learning community is invited to review and criticize this work right away for everyone to see so to me. This marks the third way in which deep learning community has been changing and evolving over time it aggressively promotes the discussion and the open criticism of deep learning results. And now in fact this approach of as soon as you have results that can be presented to put in a lock. I've and then discuss it. Openly on social media for instance is vastly adopted by deep learning researchers instead of waiting for the seal of approval from conferences and journals. So this is great for science we get to iterate over ideas much more rapidly. This is not so great for scientists because any day can be a day where you discover that some other lab has executed the research idea you wanted to work on then. In 2014 we start seeing deep learning systems. That are very good with text. So for instance we see first examples of deep learning systems successfully performing machine translation so taking a sentence in a foreign language and producing an English translation we also see systems that instead take in or reads an image and produces an English description of what that image is and this is a really interesting example because that here in a few months four different labs proposed more or less the same idea at about exactly the same time independently so this really illustrates how rapid innovation becomes at this time thanks to GPUs to graphics cards and thanks to really good open source software we get to iterate and and produce results very rapidly and then those are communicated almost immediately for everyone to digest and and and dissect laying the groundwork for the next innovation in 2015 we start seeing deep learning systems that instead of perceiving or taking as input some data and making some predictions actually can generate or synthesize visual content so have an example here of an argument the neural style transfer algorithm based on deep learning that can read a picture photograph and also a painting and then produce a painting of that photograph using the style of the painting that was provided but also we're now seeing a lot of work on generating entirely new visual content much like in this work from open AI reaching levels of realism we haven't seen before and this goes even beyond visual content where we're seeing for instance recent work by google deepmind on generating audio generating speech and generating music and also we've seen in 2016.
Perhaps you've heard about this this deep. Trump Twitter bot that's powered by deep learning where deep learning system was trained on Donald Trump's tweets and was able to generate new tweets. That might as well have come from from him now. This might make it sound easier than it was to achieve but this is actually an impressive feat but 2016 will almost certainly be remembered as the year that Google presented their alphago system and which comes PETA against one of the world's best go player Lisa doll and it won and in fact this came as a big surprise for many in the community many expected it would take many more years to actually achieve this but today alphago amongst its human peers is recognized as the second best goal player in the world so we went from deep learning systems that can take as input an image and detect simple symbols in it - deep learning systems that can both perceive and synthesize very complex content much like photographs speech text or game strategies.
So if you come a long way but there's still a long way to go before we reach tree. I and I'm quite optimistic. That deep learning will play an important role in that quest not just because deep learning technology is powerful but also and I want to leave you with this because the deep learning community has really structured itself to facilitate innovation very quickly. It has done this by first. Becoming much better exploiting computation. Resources using graphics card has become better at producing tools for performing deep learning research with very high quality open source code libraries and has become really good at discussing and sharing information about how to do deep learning and also what is the current state of the art one of the recent breakthroughs and opening up the discussion to everyone. We've got a long way in these three aspects since I've done my PhD and I think we can go even further. We're starting to see on social media. People even sharing preliminary results or early implementations of ideas or just ideas for other people perhaps to implement and so this hints at a future where different research labs might actually much more openly and collectively work to make progress towards AI. And then hopefully we'll reach a day where it actually doesn't matter at which college you decide to go or not to go to thank you you.
Now this might not be surprising that to develop an artificial intelligence that we might need an artificial brain and in fact as far back as the 1980s and even before then there were a lot of researchers performing research on designing better artificial neural networks but by the time that I started my PhD that activity had reduced quite a bit and there was only a handful of labs still performing research on artificial neural networks and the reason is that then there were a lot of other different machine learning methods that seemed more successful at simple AI tasks and in fact the research on artificial neural networks seemed to be mostly successful with simple artificial neural networks with a single layer so kind of like a brain but with just one brain region and in fact there were a lot of researchers that had essentially just given up on the artificial neural networks approach and it wouldn't be uncommon for researchers like me to submit work at conferences and get reviews at red a little bit like this where are we we just rejected just for using artificial neural networks. This isn't an exact quote. I couldn't find it back in my emails. You can probably imagine. I didn't care much for it then just got rid of it and yet now fast forward.
Ten years and deep learning is all the rage in academia. It's one of the most popular topic of research in industry deep. Learning technologies are being acquired at the millions of dollars and in the media and press. It's often reported as the new. AI much like in this piece and Scientific American so what happened well what I thought I'd do today is give you my perspective on the last ten years in deep learning that is from its emergence and how it evolved and progressed through the years. I'll talk not just about the different technology breakthroughs but also focus a bit on how the community itself evolved and progressed so for me. Things really started in 2006. The thing that really influenced my research was this paper by. Geoffrey Hinton who you see here from University of Toronto with Cimino Sendero and UOIT so in that paper Geoffrey Hinton was proposing a new approach to artificial neural networks and what was really exciting about this work is that it achieved deep artificial neural networks. That would rival some of the more standard more popular machine learning methods of the time so this really sparked a new hope in that the approach using artificial neural networks might actually be successful for achieving. AI this was a new hope so it was new so people essentially came up with a new name to refer to that type of research and they called it deep learning so for instance the next year Ike organized with some of my colleagues the first deep learning workshop it was. We tried to organise it that's part of the neural information processing systems conference which is one of the largest machine learning conferences and so we submitted a proposal for the this workshop but it was rejected however. Geoffrey Hinton just wouldn't have it so he put together the resources necessary for us to actually organize it as a parallel event and was a huge success. We attracted about 10 times as many people as other official workshops that happen during the conference so it was clear there was a lot of excitement in academia for the potential of deep artificial neural networks.
And then in the next three years we started seeing an emergence of more more papers on deep artificial neural networks referred instead under the name of deep learning now there were a lot of papers published but the progress was relatively slow. I turns out that executing artificial neural networks on regular computers is slow and so in about 2010 several different labs figured out a way of executing artificial neural networks not on standard computers but on graphics cards on GPUs the same graphics cards that we use to generate crisp graphics for computer games. So this marks for me the first way major way in which the deep learning community has been changing it has become way. Better at exploiting computational resources what this math is that a deep learning research lab could essentially build its own mini supercomputer but at a few thousand dollars and in fact it's that year that Geoffrey Hinton and his lab produced the first result suggesting that deep learning microevolution eyes speech recognition research. This came as a big surprise and in fact the speech research community a kind of difficulty believing some of these results or at least they were harder to publish initially but now deep learning is in a big way present in speech recognition research. And it's also part of the technology like behind Syrian Alexa then. In 2011 we start seeing the emergence of a lot of really good high-quality software's for in libraries for supporting deep learning research like piano and torch and a few others and to me. This marks the second way in which deep learning has really been changing over the years. It now has a new dedication towards creating high-quality robust easy-to-use open and free code libraries to support the learning research so used to be a target or neural networks. Were somewhat difficult to use and implement but now it's actually quite easy to get started by leveraging the work of other people through these open-source libraries so deep learning community has made performing deep learning research much less like carpentry and much more like playing with LEGOs in 2012 Jeff Renton Jeff fintan or prepper prepares the next revolution with deep learning this time in computer vision so him in his lab participate to a computer vision competition.
The challenge here is to design a system that can read a photograph and identify what are the objects and animals in this photograph and so the results come in and it turns out that their system totally crushes the competition and reaches accuracies. That were never seen before now. This time this breakthrough was undoubted. In fact now in computer vision it is also a field that's in large part dominated by deep learning methods right now so in 2013 there starts being a lot of excitement around deep learning methods and that excitement that year is about to transition to industry and in a big way so for instance that year with my colleagues from 2007 we decided to organize another edition of the deep learning workshop. This time our proposal is accepted and in fact not just that but we get folks from Facebook that reach out and say that their CEO Mark Zuckerberg himself actually wants to be present and participate. So let me try to convey how unusual this is organizing an academic workshop and have mark zuckerberg show up is kind of like organizing a party with your personal friends and then well look at that. Mark Zuckerberg is here. This is totally a surprise. And and not just that for someone like me who used to do research initially in my PhD and I could barely get my colleagues and other topics on machine. Learning interested in artificial neural networks is almost beyond comprehension. In fact the end the interest from industry is as high as ever and also at that workshop we see the first demonstration by little-known startup deepmind technologies of first version of their system that is able to play.
Atari games at the level of humans and in fact less than a year later the plan was acquired by Google. Also that year in 2013 the International Conference on learning representations is created I had the honour of co-chairing that conference in the past two years and I mentioned it for two reasons. The first is that this conference is now mostly known as the deep learning conference and so that means that in 2013 the community is big enough a vibrant enough than it can sustain. Its own conference. The other reason most important reason is that this conference has a very unique reviewing model for scientific work authors are asked to submit their work publicly right away on a website known as archive.org so now the work is accessible for everyone and then the whole deep learning community is invited to review and criticize this work right away for everyone to see so to me. This marks the third way in which deep learning community has been changing and evolving over time it aggressively promotes the discussion and the open criticism of deep learning results. And now in fact this approach of as soon as you have results that can be presented to put in a lock. I've and then discuss it. Openly on social media for instance is vastly adopted by deep learning researchers instead of waiting for the seal of approval from conferences and journals. So this is great for science we get to iterate over ideas much more rapidly. This is not so great for scientists because any day can be a day where you discover that some other lab has executed the research idea you wanted to work on then. In 2014 we start seeing deep learning systems. That are very good with text. So for instance we see first examples of deep learning systems successfully performing machine translation so taking a sentence in a foreign language and producing an English translation we also see systems that instead take in or reads an image and produces an English description of what that image is and this is a really interesting example because that here in a few months four different labs proposed more or less the same idea at about exactly the same time independently so this really illustrates how rapid innovation becomes at this time thanks to GPUs to graphics cards and thanks to really good open source software we get to iterate and and produce results very rapidly and then those are communicated almost immediately for everyone to digest and and and dissect laying the groundwork for the next innovation in 2015 we start seeing deep learning systems that instead of perceiving or taking as input some data and making some predictions actually can generate or synthesize visual content so have an example here of an argument the neural style transfer algorithm based on deep learning that can read a picture photograph and also a painting and then produce a painting of that photograph using the style of the painting that was provided but also we're now seeing a lot of work on generating entirely new visual content much like in this work from open AI reaching levels of realism we haven't seen before and this goes even beyond visual content where we're seeing for instance recent work by google deepmind on generating audio generating speech and generating music and also we've seen in 2016.
Perhaps you've heard about this this deep. Trump Twitter bot that's powered by deep learning where deep learning system was trained on Donald Trump's tweets and was able to generate new tweets. That might as well have come from from him now. This might make it sound easier than it was to achieve but this is actually an impressive feat but 2016 will almost certainly be remembered as the year that Google presented their alphago system and which comes PETA against one of the world's best go player Lisa doll and it won and in fact this came as a big surprise for many in the community many expected it would take many more years to actually achieve this but today alphago amongst its human peers is recognized as the second best goal player in the world so we went from deep learning systems that can take as input an image and detect simple symbols in it - deep learning systems that can both perceive and synthesize very complex content much like photographs speech text or game strategies.
So if you come a long way but there's still a long way to go before we reach tree. I and I'm quite optimistic. That deep learning will play an important role in that quest not just because deep learning technology is powerful but also and I want to leave you with this because the deep learning community has really structured itself to facilitate innovation very quickly. It has done this by first. Becoming much better exploiting computation. Resources using graphics card has become better at producing tools for performing deep learning research with very high quality open source code libraries and has become really good at discussing and sharing information about how to do deep learning and also what is the current state of the art one of the recent breakthroughs and opening up the discussion to everyone. We've got a long way in these three aspects since I've done my PhD and I think we can go even further. We're starting to see on social media. People even sharing preliminary results or early implementations of ideas or just ideas for other people perhaps to implement and so this hints at a future where different research labs might actually much more openly and collectively work to make progress towards AI. And then hopefully we'll reach a day where it actually doesn't matter at which college you decide to go or not to go to thank you you.