• <div><p>Sorry but I've added a dropout layer behind the convolution, and the memory of the GPUs explode. How could I solve this?</p><p>该提问来源于开源项目:ndrplz/ConvLSTM_pytorch</p></div>
  • dropout

    2020-12-08 19:46:23
    t look like the dropout parameter is implemented. Have you trained with dropout (as mentioned in the paper).</p><p>该提问来源于开源项目:leaderj1001/MobileNetV3-Pytorch</p></div>
  • Dropout

    2020-11-26 21:52:40
    <div><p>I would like to have your thoughts about the best way to implement Dropout in blocks. In my opinion there are two options: - Suggested by : After construction of the model, manipulate the ...
  • <p>When trying to apply dropout on the rnn encoder, I get this warning: <pre><code> /lib/python3.6/site-packages/torch/nn/modules/rnn.py:50: UserWarning: dropout option adds dropout after all but last...
  • This is a very good paper talking about the fundamental of dropout mechanism
  • 论文链接: 本文提出一个新奇的方法来加快神经网络的训练速度, 基于... 然而进行独立成分分析是十分耗时的, 为此作者提出一种被称为 Independent-Component (IC) 的神经网络结构来实现, 这个结构是批归一化和 dropout...

    本文提出一个新奇的方法来加快神经网络的训练速度, 基于这样一个想法, 即对输入进行白化处理有助于加快收敛速度. 已知独立的随机变量之间一定是白化的(不相关的), 所以对网络输入进行独立成分分析能够获得白化的输入. 然而进行独立成分分析是十分耗时的, 为此作者提出一种被称为 Independent-Component (IC) 的神经网络结构来实现, 这个结构是批归一化和 dropout 的组合. 本文通过理论分析了这一结构的有效性, 通过实验也表明了这一改进提升了分类性能.

    论文链接: https://arxiv.org/abs/1905.05928v1


    白化是一种用于去除数据之间相关性的方法, 这使得网络训练更加容易 (Le Cun et al., 1991), 但 Le Cun 所做的工作仅限于预处理训练数据. 而在神经网络中每个激活层上实现白化则需要大量的计算, 因此有了后来的批归一化方法(Batch Normalization, (Ioffe & Szegedy, 2015)), 该方法通过将每个激活层的净激活比例缩放为零均值和单位方差来增强性能. 批归一化程序可以显着平滑参数空间中的优化环境 (Bjorck et al., 2018),以提高训练性能. 由于批归一化易于实施和成功. 因此注意力从最初的白化目标转移了. 这项工作的动机是恢复对设计出高效的计算方法来白化每一层的输入的追求.

    为什么要通过获得变量的独立成分来进行白化呢? 这受到神经科学的一项发现的支持: 神经系统的表达能力随群体中独立神经元的数量而线性增加 (Moreno-Bote et al., 2014; Beaulieu-Laroche et al., 2018). 因此要使得每一层的输入变量之间尽可能独立, 这也将使得训练更加稳定. 本文提出的 IC 层是由批归一化 Batch Normalization (Ioffe & Szegedy, 2015) 和 Dropout (Srivastava et al., 2014) 组成的, 这两种方法能够很容易的实现.

    在 (Huang et al., 2018) 中提出了在网络中添加额外的层来实现独立成分分析, 然而这种方法计算两太大了.


    本文提出的 IC 层结构是非常简单的, 首先它的位置如下图所示:


    IC 层位于激活层之后, 它的下一层就是卷积层. IC 层的结构从代码上来看如下:


    也就是简单得将 BatchNorm 和 Dropout 叠加在一起.

    IC 层能够二次方地减少变量之间的非独立性, 线性地减少相关性. 在减少非独立性的同时, Dropout 操作也会线性的减少信息熵, 作者提到可以通过调整 pp 值来权衡获得的独立性和减少的信息熵. 有关理论证明请查看原文.

    IC 方法直观解释如下: 批归一化使得变量具有零均值和单位方差, 这 ZCA 方法一样.Dropout 操作使得神经元以概率 p 输出其值, 或者通过输出零来使神经元失活,
    这样一来经元的输出的信息将很少来自其他神经元, 也就是使得这些神经元在统计上变得彼此独立.


    • Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
    • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
    • Le Cun, Y., Kanter, I., and Solla, S. A. Eigenvalues of covariance matrices: Application to neural-network learning. Physical Review Letters, 66(18):2396, 1991.
    • Moreno-Bote, R., Beck, J., Kanitscheider, I., Pitkow, X., Latham, P., and Pouget, A. Information-limiting correlations. Nature neuroscience, 17(10):1410, 2014.
    • Beaulieu-Laroche, L., Toloza, E. H., van der Goes, M.-S., Lafourcade, M., Barnagian, D., Williams, Z. M., Eskandar, E. N., Frosch, M. P., Cash, S. S., and Harnett, M. T. Enhanced dendritic compartmentalization in human cortical neurons. Cell, 175(3):643–651, 2018.


    互信息可用来衡量变量之间的独立性, 而相关性则用协方差来表示.

    独立成分分析 (Independent Component Analysis, ICA)

    , 论文: Independent Component Analysis: Algorithms and Applications




    Whitening (白化)



    为什么使用白化? 教程给出的解释是:



    注意: 白化可以去除相关性, 但不能去除非独立性, 即独立不一定相关, 但是不相关不一定独立. 详见 https://blog.csdn.net/Tonywu2018/article/details/91490158

  • Chapter 3: The Dropout Turn On, Tune In . . . Chrisann Brennan Toward the end of his senior year at Homestead, in the spring of 1972, Jobs started going out with a girl named Chrisann Brennan, who was...

    Chapter 3: The Dropout

    Turn On, Tune In . . .

    Chrisann Brennan

    Toward the end of his senior year at Homestead, in the spring of 1972, Jobs started going out with a girl named Chrisann Brennan, who was about his age but still a junior. With her light brown hair, green eyes, high cheekbones, and fragile aura, she was very attractive. She was also enduring the breakup of her parents’ marriage, which made her vulnerable. “We worked together on an animated movie, then started going out, and she became my first real girlfriend,” Jobs recalled. As Brennan later said, “Steve was kind of crazy. That’s why I was attracted to him.”

    Jobs’s craziness was of the cultivated sort. He had begun his lifelong experiments with compulsive diets, eating only fruits and vegetables, so he was as lean and tight as a whippet. He learned to stare at people without blinking, and he perfected long silences punctuated by staccato bursts of fast talking. This odd mix of intensity and aloofness, combined with his shoulder-length hair and scraggly beard, gave him the aura of a crazed shaman. He oscillated between charismatic and creepy. “He shuffled around and looked half-mad,” recalled Brennan. “He had a lot of angst. It was like a big darkness around him.”

    Jobs had begun to drop acid by then, and he turned Brennan on to it as well, in a wheat field just outside Sunnyvale. “It was great,” he recalled. “I had been listening to a lot of Bach. All of a sudden the wheat field was playing Bach. It was the most wonderful feeling of my life up to that point. I felt like the conductor of this symphony with Bach coming through the wheat.”

    That summer of 1972, after his graduation, he and Brennan moved to a cabin in the hills above Los Altos. “I’m going to go live in a cabin with Chrisann,” he announced to his parents one day. His father was furious. “No you’re not,” he said. “Over my dead body.” They had recently fought about marijuana, and once again the younger Jobs was willful. He just said good-bye and walked out.

    Brennan spent a lot of her time that summer painting; she was talented, and she did a picture of a clown for Jobs that he kept on the wall. Jobs wrote poetry and played guitar. He could be brutally cold and rude to her at times, but he was also entrancing and able to impose his will. “He was an enlightened being who was cruel,” she recalled. “That’s a strange combination.”

    Midway through the summer, Jobs was almost killed when his red Fiat caught fire. He was driving on Skyline Boulevard in the Santa Cruz Mountains with a high school friend, Tim Brown, who looked back, saw flames coming from the engine, and casually said to Jobs, “Pull over, your car is on fire.” Jobs did. His father, despite their arguments, drove out to the hills to tow the Fiat home.

    In order to find a way to make money for a new car, Jobs got Wozniak to drive him to De Anza College to look on the help-wanted bulletin board. They discovered that the Westgate Shopping Center in San Jose was seeking college students who could dress up in costumes and amuse the kids. So for $3 an hour, Jobs, Wozniak, and Brennan donned heavy full-body costumes and headgear to play Alice in Wonderland, the Mad Hatter, and the White Rabbit. Wozniak, in his earnest and sweet way, found it fun. “I said, ‘I want to do it, it’s my chance, because I love children.’ I think Steve looked at it as a lousy job, but I looked at it as a fun adventure.” Jobs did indeed find it a pain. “It was hot, the costumes were heavy, and after a while I felt like I wanted to smack some of the kids.” Patience was never one of his virtues.

    Reed College

    Seventeen years earlier, Jobs’s parents had made a pledge when they adopted him: He would go to college. So they had worked hard and saved dutifully for his college fund, which was modest but adequate by the time he graduated. But Jobs, becoming ever more willful, did not make it easy. At first he toyed with not going to college at all. “I think I might have headed to New York if I didn’t go to college,” he recalled, musing on how different his world—and perhaps all of ours—might have been if he had chosen that path. When his parents pushed him to go to college, he responded in a passive-aggressive way. He did not consider state schools, such as Berkeley, where Woz then was, despite the fact that they were more affordable. Nor did he look at Stanford, just up the road and likely to offer a scholarship. “The kids who went to Stanford, they already knew what they wanted to do,” he said. “They weren’t really artistic. I wanted something that was more artistic and interesting.”

    Instead he insisted on applying only to Reed College, a private liberal arts school in Portland, Oregon, that was one of the most expensive in the nation. He was visiting Woz at Berkeley when his father called to say an acceptance letter had arrived from Reed, and he tried to talk Steve out of going there. So did his mother. It was far more than they could afford, they said. But their son responded with an ultimatum: If he couldn’t go to Reed, he wouldn’t go anywhere. They relented, as usual.

    Reed had only one thousand students, half the number at Homestead High. It was known for its free-spirited hippie lifestyle, which combined somewhat uneasily with its rigorous academic standards and core curriculum. Five years earlier Timothy Leary, the guru of psychedelic enlightenment, had sat cross-legged at the Reed College commons while on his League for Spiritual Discovery (LSD) college tour, during which he exhorted his listeners, “Like every great religion of the past we seek to find the divinity within. . . . These ancient goals we define in the metaphor of the present—turn on, tune in, drop out.” Many of Reed’s students took all three of those injunctions seriously; the dropout rate during the 1970s was more than one-third.

    When it came time for Jobs to matriculate in the fall of 1972, his parents drove him up to Portland, but in another small act of rebellion he refused to let them come on campus. In fact he refrained from even saying good-bye or thanks. He recounted the moment later with uncharacteristic regret:

    It’s one of the things in life I really feel ashamed about. I was not very sensitive, and I hurt their feelings. I shouldn’t have. They had done so much to make sure I could go there, but I just didn’t want them around. I didn’t want anyone to know I had parents. I wanted to be like an orphan who had bummed around the country on trains and just arrived out of nowhere, with no roots, no connections, no background.

    In late 1972, there was a fundamental shift happening in American campus life. The nation’s involvement in the Vietnam War, and the draft that accompanied it, was winding down. Political activism at colleges receded and in many late-night dorm conversations was replaced by an interest in pathways to personal fulfillment. Jobs found himself deeply influenced by a variety of books on spirituality and enlightenment, most notably Be Here Now, a guide to meditation and the wonders of psychedelic drugs by Baba Ram Dass, born Richard Alpert. “It was profound,” Jobs said. “It transformed me and many of my friends.”

    The closest of those friends was another wispy-bearded freshman named Daniel Kottke, who met Jobs a week after they arrived at Reed and shared his interest in Zen, Dylan, and acid. Kottke, from a wealthy New York suburb, was smart but low-octane, with a sweet flower-child demeanor made even mellower by his interest in Buddhism. That spiritual quest had caused him to eschew material possessions, but he was nonetheless impressed by Jobs’s tape deck. “Steve had a TEAC reel-to-reel and massive quantities of Dylan bootlegs,” Kottke recalled. “He was both really cool and high-tech.”

    Jobs started spending much of his time with Kottke and his girlfriend, Elizabeth Holmes, even after he insulted her at their first meeting by grilling her about how much money it would take to get her to have sex with another man. They hitchhiked to the coast together, engaged in the typical dorm raps about the meaning of life, attended the love festivals at the local Hare Krishna temple, and went to the Zen center for free vegetarian meals. “It was a lot of fun,” said Kottke, “but also philosophical, and we took Zen very seriously.”

    Jobs began sharing with Kottke other books, including Zen Mind, Beginner’s Mind by Shunryu Suzuki, Autobiography of a Yogi by Paramahansa Yogananda, and Cutting Through Spiritual Materialism by Ch?gyam Trungpa. They created a meditation room in the attic crawl space above Elizabeth Holmes’s room and fixed it up with Indian prints, a dhurrie rug, candles, incense, and meditation cushions. “There was a hatch in the ceiling leading to an attic which had a huge amount of space,” Jobs said. “We took psychedelic drugs there sometimes, but mainly we just meditated.”

    Jobs’s engagement with Eastern spirituality, and especially Zen Buddhism, was not just some passing fancy or youthful dabbling. He embraced it with his typical intensity, and it became deeply ingrained in his personality. “Steve is very much Zen,” said Kottke. “It was a deep influence. You see it in his whole approach of stark, minimalist aesthetics, intense focus.” Jobs also became deeply influenced by the emphasis that Buddhism places on intuition. “I began to realize that an intuitive understanding and consciousness was more significant than abstract thinking and intellectual logical analysis,” he later said. His intensity, however, made it difficult for him to achieve inner peace; his Zen awareness was not accompanied by an excess of calm, peace of mind, or interpersonal mellowness.

    He and Kottke enjoyed playing a nineteenth-century German variant of chess called Kriegspiel, in which the players sit back-to-back; each has his own board and pieces and cannot see those of his opponent. A moderator informs them if a move they want to make is legal or illegal, and they have to try to figure out where their opponent’s pieces are. “The wildest game I played with them was during a lashing rainstorm sitting by the fireside,” recalled Holmes, who served as moderator. “They were tripping on acid. They were moving so fast I could barely keep up with them.”

    Another book that deeply influenced Jobs during his freshman year was Diet for a Small Planet by Frances Moore Lappé, which extolled the personal and planetary benefits of vegetarianism. “That’s when I swore off meat pretty much for good,” he recalled. But the book also reinforced his tendency to embrace extreme diets, which included purges, fasts, or eating only one or two foods, such as carrots or apples, for weeks on end.

    Jobs and Kottke became serious vegetarians during their freshman year. “Steve got into it even more than I did,” said Kottke. “He was living off Roman Meal cereal.” They would go shopping at a farmers’ co-op, where Jobs would buy a box of cereal, which would last a week, and other bulk health food. “He would buy flats of dates and almonds and lots of carrots, and he got a Champion juicer and we’d make carrot juice and carrot salads. There is a story about Steve turning orange from eating so many carrots, and there is some truth to that.” Friends remember him having, at times, a sunset-like orange hue.

    Jobs’s dietary habits became even more obsessive when he read Mucusless Diet Healing System by Arnold Ehret, an early twentieth-century German-born nutrition fanatic. He believed in eating nothing but fruits and starchless vegetables, which he said prevented the body from forming harmful mucus, and he advocated cleansing the body regularly through prolonged fasts. That meant the end of even Roman Meal cereal—or any bread, grains, or milk. Jobs began warning friends of the mucus dangers lurking in their bagels. “I got into it in my typical nutso way,” he said. At one point he and Kottke went for an entire week eating only apples, and then Jobs began to try even purer fasts. He started with two-day fasts, and eventually tried to stretch them to a week or more, breaking them carefully with large amounts of water and leafy vegetables. “After a week you start to feel fantastic,” he said. “You get a ton of vitality from not having to digest all this food. I was in great shape. I felt I could get up and walk to San Francisco anytime I wanted.”

    Vegetarianism and Zen Buddhism, meditation and spirituality, acid and rock—Jobs rolled together, in an amped-up way, the multiple impulses that were hallmarks of the enlightenment-seeking campus subculture of the era. And even though he barely indulged it at Reed, there was still an undercurrent of electronic geekiness in his soul that would someday combine surprisingly well with the rest of the mix.

    Robert Friedland

    In order to raise some cash one day, Jobs decided to sell his IBM Selectric typewriter. He walked into the room of the student who had offered to buy it only to discover that he was having sex with his girlfriend. Jobs started to leave, but the student invited him to take a seat and wait while they finished. “I thought, ‘This is kind of far out,’” Jobs later recalled. And thus began his relationship with Robert Friedland, one of the few people in Jobs’s life who were able to mesmerize him. He adopted some of Friedland’s charismatic traits and for a few years treated him almost like a guru—until he began to see him as a charlatan.

    Friedland was four years older than Jobs, but still an undergraduate. The son of an Auschwitz survivor who became a prosperous Chicago architect, he had originally gone to Bowdoin, a liberal arts college in Maine. But while a sophomore, he was arrested for possession of 24,000 tablets of LSD worth $125,000. The local newspaper pictured him with shoulder-length wavy blond hair smiling at the photographers as he was led away. He was sentenced to two years at a federal prison in Virginia, from which he was paroled in 1972. That fall he headed off to Reed, where he immediately ran for student body president, saying that he needed to clear his name from the “miscarriage of justice” he had suffered. He won.

    Friedland had heard Baba Ram Dass, the author of Be Here Now, give a speech in Boston, and like Jobs and Kottke had gotten deeply into Eastern spirituality. During the summer of 1973, he traveled to India to meet Ram Dass’s Hindu guru, Neem Karoli Baba, famously known to his many followers as Maharaj-ji. When he returned that fall, Friedland had taken a spiritual name and walked around in sandals and flowing Indian robes. He had a room off campus, above a garage, and Jobs would go there many afternoons to seek him out. He was entranced by the apparent intensity of Friedland’s conviction that a state of enlightenment truly existed and could be attained. “He turned me on to a different level of consciousness,” Jobs said.

    Friedland found Jobs fascinating as well. “He was always walking around barefoot,” he later told a reporter. “The thing that struck me was his intensity. Whatever he was interested in he would generally carry to an irrational extreme.” Jobs had honed his trick of using stares and silences to master other people. “One of his numbers was to stare at the person he was talking to. He would stare into their fucking eyeballs, ask some question, and would want a response without the other person averting their eyes.”

    According to Kottke, some of Jobs’s personality traits—including a few that lasted throughout his career—were borrowed from Friedland. “Friedland taught Steve the reality distortion field,” said Kottke. “He was charismatic and a bit of a con man and could bend situations to his very strong will. He was mercurial, sure of himself, a little dictatorial. Steve admired that, and he became more like that after spending time with Robert.”

    Jobs also absorbed how Friedland made himself the center of attention. “Robert was very much an outgoing, charismatic guy, a real salesman,” Kottke recalled. “When I first met Steve he was shy and self-effacing, a very private guy. I think Robert taught him a lot about selling, about coming out of his shell, of opening up and taking charge of a situation.” Friedland projected a high-wattage aura. “He would walk into a room and you would instantly notice him. Steve was the absolute opposite when he came to Reed. After he spent time with Robert, some of it started to rub off.”

    On Sunday evenings Jobs and Friedland would go to the Hare Krishna temple on the western edge of Portland, often with Kottke and Holmes in tow. They would dance and sing songs at the top of their lungs. “We would work ourselves into an ecstatic frenzy,” Holmes recalled. “Robert would go insane and dance like crazy. Steve was more subdued, as if he was embarrassed to let loose.” Then they would be treated to paper plates piled high with vegetarian food.

    Friedland had stewardship of a 220-acre apple farm, about forty miles southwest of Portland, that was owned by an eccentric millionaire uncle from Switzerland named Marcel Müller. After Friedland became involved with Eastern spirituality, he turned it into a commune called the All One Farm, and Jobs would spend weekends there with Kottke, Holmes, and like-minded seekers of enlightenment. The farm had a main house, a large barn, and a garden shed, where Kottke and Holmes slept. Jobs took on the task of pruning the Gravenstein apple trees. “Steve ran the apple orchard,” said Friedland. “We were in the organic cider business. Steve’s job was to lead a crew of freaks to prune the orchard and whip it back into shape.”

    Monks and disciples from the Hare Krishna temple would come and prepare vegetarian feasts redolent of cumin, coriander, and turmeric. “Steve would be starving when he arrived, and he would stuff himself,” Holmes recalled. “Then he would go and purge. For years I thought he was bulimic. It was very upsetting, because we had gone to all that trouble of creating these feasts, and he couldn’t hold it down.”

    Jobs was also beginning to have a little trouble stomaching Friedland’s cult leader style. “Perhaps he saw a little bit too much of Robert in himself,” said Kottke. Although the commune was supposed to be a refuge from materialism, Friedland began operating it more as a business; his followers were told to chop and sell firewood, make apple presses and wood stoves, and engage in other commercial endeavors for which they were not paid. One night Jobs slept under the table in the kitchen and was amused to notice that people kept coming in and stealing each other’s food from the refrigerator. Communal economics were not for him. “It started to get very materialistic,” Jobs recalled. “Everybody got the idea they were working very hard for Robert’s farm, and one by one they started to leave. I got pretty sick of it.”

    Many years later, after Friedland had become a billionaire copper and gold mining executive—working out of Vancouver, Singapore, and Mongolia—I met him for drinks in New York. That evening I emailed Jobs and mentioned my encounter. He telephoned me from California within an hour and warned me against listening to Friedland. He said that when Friedland was in trouble because of environmental abuses committed by some of his mines, he had tried to contact Jobs to intervene with Bill Clinton, but Jobs had not responded. “Robert always portrayed himself as a spiritual person, but he crossed the line from being charismatic to being a con man,” Jobs said. “It was a strange thing to have one of the spiritual people in your young life turn out to be, symbolically and in reality, a gold miner.”

    . . . drop Out

    Jobs quickly became bored with college. He liked being at Reed, just not taking the required classes. In fact he was surprised when he found out that, for all of its hippie aura, there were strict course requirements. When Wozniak came to visit, Jobs waved his schedule at him and complained, “They are making me take all these courses.” Woz replied, “Yes, that’s what they do in college.” Jobs refused to go to the classes he was assigned and instead went to the ones he wanted, such as a dance class where he could enjoy both the creativity and the chance to meet girls. “I would never have refused to take the courses you were supposed to, that’s a difference in our personality,” Wozniak marveled.

    Jobs also began to feel guilty, he later said, about spending so much of his parents’ money on an education that did not seem worthwhile. “All of my working-class parents’ savings were being spent on my college tuition,” he recounted in a famous commencement address at Stanford. “I had no idea what I wanted to do with my life and no idea how college was going to help me figure it out. And here I was spending all of the money my parents had saved their entire life. So I decided to drop out and trust that it would all work out okay.”

    He didn’t actually want to leave Reed; he just wanted to quit paying tuition and taking classes that didn’t interest him. Remarkably, Reed tolerated that. “He had a very inquiring mind that was enormously attractive,” said the dean of students, Jack Dudman. “He refused to accept automatically received truths, and he wanted to examine everything himself.” Dudman allowed Jobs to audit classes and stay with friends in the dorms even after he stopped paying tuition.

    “The minute I dropped out I could stop taking the required classes that didn’t interest me, and begin dropping in on the ones that looked interesting,” he said. Among them was a calligraphy class that appealed to him after he saw posters on campus that were beautifully drawn. “I learned about serif and sans serif typefaces, about varying the amount of space between different letter combinations, about what makes great typography great. It was beautiful, historical, artistically subtle in a way that science can’t capture, and I found it fascinating.”

    It was yet another example of Jobs consciously positioning himself at the intersection of the arts and technology. In all of his products, technology would be married to great design, elegance, human touches, and even romance. He would be in the fore of pushing friendly graphical user interfaces. The calligraphy course would become iconic in that regard. “If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, it’s likely that no personal computer would have them.”

    In the meantime Jobs eked out a bohemian existence on the fringes of Reed. He went barefoot most of the time, wearing sandals when it snowed. Elizabeth Holmes made meals for him, trying to keep up with his obsessive diets. He returned soda bottles for spare change, continued his treks to the free Sunday dinners at the Hare Krishna temple, and wore a down jacket in the heatless garage apartment he rented for $20 a month. When he needed money, he found work at the psychology department lab maintaining the electronic equipment that was used for animal behavior experiments. Occasionally Chrisann Brennan would come to visit. Their relationship sputtered along erratically. But mostly he tended to the stirrings of his own soul and personal quest for enlightenment.

    “I came of age at a magical time,” he reflected later. “Our consciousness was raised by Zen, and also by LSD.” Even later in life he would credit psychedelic drugs for making him more enlightened. “Taking LSD was a profound experience, one of the most important things in my life. LSD shows you that there’s another side to the coin, and you can’t remember it when it wears off, but you know it. It reinforced my sense of what was important—creating great things instead of making money, putting things back into the stream of history and of human consciousness as much as I could.”

  • <div><p>该提问来源于开源项目:amazon-archives/amazon-dsstne</p></div>
  • added dropout

    2021-01-10 07:10:11
    <div><p>Added an example code of dropout over the existing MLP and Logistic Regression tutorial code. The code that i have attached is the full working version. It does reuse some modules of the ...
  • youtu.be">video </a>of <a href="http://cs231n.stanford.edu/syllabus.html">cs231n</a> at 51:30, the activation of the dropout should be rescaled, so the output at text time == expected output ...
  • <div><p>I have tested that with both of the bugs fixed, good alignments can be learned with reduction factor 2 easily.</p><p>该提问来源于开源项目:keithito/tacotron</p></div>
  • <div><p>Is the <code>--sentencepiece-alphas</code> in Marian CLI the same as the alpha on https://github.com/google/sentencepiece/blob/master/src/bpe_model.h#L43 to support BPE dropout when called at ...
  • About Dropout

    2020-12-08 23:29:30
    You use dropout in the 2 fc_layer in your caffe version code <a href="https://github.com/ppengtang/oicr/blob/pcl-multiGPU/models/VGG16/train.prototxt">here</a> , but not here, does it work for the ...
  • I can see from the document that the DROPOUT FROM GcBiasMetrics is about the percentage of misaligned reads that correlate with low (%-GC is < 50%) or high (%-GC is > 50%) GC content ...
  • Dropout fix 2

    2021-01-09 19:24:31
    I replaced the dropout probability by the keep probability and inverted the condition so the number of inputs that drop out stays the same. The result is that <code>HadamardDiv</code> at the end of ...
  • Dropout speed issues

    2021-01-08 11:44:35
    The only change in my code was do use dropout in the hidden layers of the net. Also, GPU usage is now way down, and a lot of time it actually reaches 0, which didn't happen before. <p>Any clues?...
  • Eliminate dropout?

    2020-12-06 23:55:05
    For example, you might want to freeze the graph to deploy the .pb file in iOS/Android, which means you have to remove the dropout nodes (or get a op RandomUniform not supported error...), however, in...
  • Hilach/dropout

    2020-11-22 19:42:49
    Dropout, when used, is applied after every dense layer except for the the qkv-projections and directly after adding positional- to patch embeddings. So 3 things regarding the dropouts you added: 1. I...
  • 首先大家都知道自BN提出之后,Dropout突然失宠,究其原因就是BN原作者提到在他们的实践中对有BN的网络加了Dropout通常性能不升反降。他们猜测BN由于引入了不同样本之间的统计量,可能产生了一定的正则化效果从而替代...






    首先大家都知道自BN提出之后,Dropout突然失宠,究其原因就是BN原作者提到在他们的实践中对有BN的网络加了Dropout通常性能不升反降。他们猜测BN由于引入了不同样本之间的统计量,可能产生了一定的正则化效果从而替代掉了Dropout的增益。之后各大流行网络ResNet,PreResNet,ResNeXt,DenseNet相继提出,大家也心照不宣地都默认缺省了Dropout。比较有意思的是Wide ResNet(WRN)的作者多了一个“心眼”:他发现在很宽的WRN网络里面,在每一个bottleneck的两个conv层之间加上那么一个Dropout,竟然能得到稳定的提升。这就显得很奇怪,也是这篇文章想要通过一些推导想搞清楚的。


    究其核心原因即是Dropout在网络测试的时候神经元会产生“variance shift”,我们将其称为方差偏移。而只要你在网络的bottleneck内部使用Dropout,在随后一定会跟上一个BN层(BN层处处都有而且一个bottleneck通常还不止一个)。BN层在test的时候,会对神经元首先施加一个归一化的操作,这个操作涉及到除以该神经元在training时候统计的方差(其实是方差的根号,我们简化了一下)。这个时候问题就来了,这个training时候统计的方差和test的时候的方差就对应了上面所说的偏移,换句话说,BN这个时候除错了!正所谓,一步错,步步错,一失足成千古恨,随着网络feature的不断forward,最终可能导致final的prediction产生了错误的偏差,从而引起性能的下降。


    于是作者就按两种情况进行分析,第一种比较极端,Dropout直接跟BN;第二种实际上就是WRN里面的情况,Dropout跟一个参数层(卷积/全连接)再跟BN。经过一番公式轰炸,在第二种情况下,从最后variance shift的表达式来看,的确增加feature的dimension是可以缓解这个偏移的。这样就很好解释了第一段提到的各种情况。


    然后就是通过实验统计看看实际情况是不是的确符合。首先,作者主要统计了CIFAR10/100上两种Dropout放置情况下,不同的drop ratio的一坨图表,而且给出了每个模型的性能对照。总体性能上讲,WRN的error柱状图的确就和其它的网络不同,通常会降个一会再升上去。这些性能有提升的带Dropout的WRN模型对应的variance
    shift曲线就非常贴近不使用Dropout的WRN时候的曲线(Figure 4的行3列4最为明显)。其它模型有些variance shift都飞到天上去了。当然,当drop ratio 很小的时候比如0.1,大家的variance shift都不会太大,所以性能上也都说的过去。


    为了进一步地说明,作者发现第一种情况下Dropout 0.5的时候,对于DenseNet和PreResNet,就连training data都中枪了。本来训练的时候分的好好的,一测试,尽管网络参数都没变,就挂了。为了直观地看一下是怎么挂的,作者从cifar10里面选了些训练正确,但测试错误的图,分别输出了一下softmax之前的neural values,的确是各有各的挂法,挂成什么类都可能。这么一来,既然我其它参数都没变,怪就只能怪“variance shift”了。


    然后又进一步,其它参数还是不变,我们来调整一下BN的这些用来测试的统计均值和统计方差(方差错了之后随后的均值也都跟着不对了,所以要一起改)试试?于是就拿training data又过了一遍网络,不过我们这次不训练,就调整一下所有BN层的统计均值和方差。然后再拿来测。果不其然,所有模型的性能都上来了,一大波挂掉的样本又复活了。


    现在我们很清楚了,“variance shift”是导致Dropout和BN合作不愉快的罪魁祸首。那么怎么搞才能让他俩一起愉快而稳定地玩耍呢?


    第一个方案比较简单,不过在现代主流网络的实现里面似乎都没有了,记得BN没出来以前好像大家也都这么做过,就是拒绝“variance shift”,把Dropout放在所有BN层的后面就完事了。鉴于现在所有BN层之后其实没什么层了,基本就到输出了,所以统一放在softmax前的最后一层上。对于CIFAR10总体比较有效,在CIFAR100上较小的drop ratio起到了作用。另外作者发现去年ImageNet的冠军SENet里面也用到了这个技巧,他们在最后一层采用了0.2的Drop ratio,作者做了测试,的确是有接近0.2个点的稳定提升。 顺带把ResNeXt和ResNet-200这几个也做了,同样也很稳定。



  • Implement dropout

    2020-12-08 19:00:14
    * [x] expand the cudnn bindings in <code>rcudnn * [x] use the <code>rcudnn</code> bindings in <code>coaster-nn * [x] create a apropriate interface in <code>coaster * [x] use that interface to define a...
  • tf.layers.dropout 和tf.nn.dropout区别

    千次阅读 2018-06-05 16:58:11
    tf.layers.dropout 和tf.nn.dropout...The only differences in the two functions are tf.nn.dropout has parameter keep_prob: “Probability that each element is kept” tf.layers.dropout has parameter rat...

    tf.layers.dropout 和tf.nn.dropout区别

    The only differences in the two functions are

    • tf.nn.dropout has parameter keep_prob: “Probability that each element is kept”
    • tf.layers.dropout has parameter rate: “The dropout rate”
      Thus, keep_prob = 1 - rate as defined here
      The tf.layers.dropout has training parameter: “Whether to return the output in training mode (apply dropout) or in inference mode (return the input untouched).”

    tf.layers 是高层API,tf.nn是低层API

    tf.layers is a higher-level wrapper, and tf.nn.dropout is from TensorFlow’s low-level library. tf.nn.dropout is there since the first public release of TensorFlow (version 0.6?), while tf.layers.dropout is there since about version 1.0 or so. As far as I know, the community develops new cool stuff in tf.contrib, which interfaces are likely to change. After a while, these are then transfered to tf.layers as soon as the interfaces (params, param-names, etc) are stable


  • <div><p>The <code>K.learning_phase()</code> flag was not being passed correctly to <code>k.function(), resulting in <code>AttributeError: 'Tensor' object has no attribute '_keras_shape'...
  • <div><p>Python Dropout op uses the following code to check keep_prob value: if tensor_util.constant_value(keep_prob) == 1: return x If keep_prob is placeholder, tensor_util.constant_value(keep...
  • Seeding Random dropout

    2020-11-27 01:50:42
    <div><p>Hello, is it possible to seed (initialize) the random dropout regularization in the MLP classifier? Thanks</p><p>该提问来源于开源项目:aigamedev/scikit-neuralnetwork</p></div>
  • Default dropout behavior

    2020-12-08 20:45:21
    <div><p>I have a couple of queries regarding the default dropout behavior in returnn. Does it also apply to recurrent weights? Does it support variational dropout, if yes then how to use it?</p><p>该...
  • <div><p>According to the papar, "We do this by randomly <strong>replacing</strong> some fraction of the conditioned-on <strong>word tokens</strong> with the generic unknown word token unk", ...
  • Sina Weibo:东莞小锋子Sexyphone Tencent E-mail:403568338@qq.com ... ... 这个Dropout技术,念成「draw泡」,主要解决神经网络的过拟合问题,提升泛化能力。  我并不打算对其进行解说,因为相关



1 2 3 4 5 ... 20
收藏数 1,650
精华内容 660