精华内容
下载资源
问答
  • qt.network.ssl: QSslSocket::connectToHostEncrypted: TLS initialization failed     网上很多说法是libeay32.dll和ssleay32.dll复制到可执行文件里,这个只是解决在Qt Creator下编译运行可以,单独打包不运行...
  • gtk initialization failed可能的解决方法之一-附件资源
  • 使用 QT5.12.5 版本,进行 HTTPS 请求时,出现 TLS initialization failed,缺少的 OpenSSL 的库
  • JUnit报initializationError

    2019-05-01 01:41:19
    NULL 博文链接:https://1960370817.iteye.com/blog/2401053
  • we build on the initialization method proposed by Martinelli [1] and extended by Kaiser et al. [2], modifying it to be more general and efficient. We improve accuracy with several rounds of visual-...
  • 但自从Windows 2012出来之后,这部分ApplicationInitialization功能已经有内含在IIS8之中,可以直接进行设定就可以。不过微软也发布了针对针对前期的IIS单独模块:ApplicationInitialization Module for IIS 7.5 。
  • ApplicationInitialization让IIS 7 如同IIS 8 第一次请求不变慢
  • Monocular Visual-Inertial State Estimation With Online Initialization and Camera-IMU Extinsic Calibration
  • ApplicationInitialization for UI x64 ApplicationInitialization for UI x64
  • 解决:oracle_plsql连服务端时 Initialization error could initialize错误处理方法
  • junit4单元测试报错:method initializationerror not found 缺少jar包导致,总共需要三个jar包:junit-4.11.jar,hamcrest-core-1.3.rc2.jar,hamcrest-library-1.3.rc2.jar,都在压缩包里了。
  • Junit出现InitializationError,原因是少了两个jar包,下载下来导入即可
  • ApplicationInitialization for UI x64
  • silverlightToolkit4 中经常缺少的DLL文件 System.ComponentModel.Composition.Initialization.dll
  • 初始化Initialization

    2019-07-22 10:41:01
    初始化Initialization初始化Initialization初始化Initialization
  • Ewf Initialization.exe

    2019-06-11 21:30:45
    很多人在使用wes7时都会讲自己的应用程序安装完成后再进行制作镜像文件,这样就会使得WES7本身自带的EWF磁盘保护功能失效,如果通过修改硬盘ID的方式虽然可以解决这个问题,但是每次都需要查询硬盘ID非常的繁琐,这...
  • CentOS6.5配置eth0重启报错提示Device eth0 does not seem to be present,delaying initialization的结决方法 在搭建Hadooop 完全分布式的时候,从节点一般是从主节点复制过来的,再给从节点配置网络的时候,会出现...
  • 吴恩达第二课第一周编程作业Initialization全代码
  • 这是我翻译的DRLSE模型,可能有少许语句不顺,但是完全不影响阅读
  • 本文节选自吴恩达老师《深度学习专项课程》编程作业,在此表示感谢。 课程链接:... 目录 1 - Neural Network model 2 - Zero initialization 3 - Random initialization(掌握) 4 - He initial...

    本文节选自吴恩达老师《深度学习专项课程》编程作业,在此表示感谢。

    课程链接:https://www.deeplearning.ai/deep-learning-specialization/

    目录

    1 - Neural Network model

    2 - Zero initialization

    3 - Random initialization(掌握)

    4 - He initialization(理解)


    To get started, run the following cell to load the packages and the planar dataset you will try to classify.

    import numpy as np
    import matplotlib.pyplot as plt
    import sklearn
    import sklearn.datasets
    from init_utils import sigmoid, relu, compute_loss, forward_propagation, backward_propagation
    from init_utils import update_parameters, predict, load_dataset, plot_decision_boundary, predict_dec
    
    %matplotlib inline
    plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
    plt.rcParams['image.interpolation'] = 'nearest'
    plt.rcParams['image.cmap'] = 'gray'
    
    # load image dataset: blue/red dots in circles
    train_X, train_Y, test_X, test_Y = load_dataset()


    1 - Neural Network model

    You will use a 3-layer neural network (already implemented for you). Here are the initialization methods you will experiment with:

    • Zeros initialization -- setting initialization = "zeros" in the input argument.
    • Random initialization -- setting initialization = "random" in the input argument. This initializes the weights to large random values.
    • He initialization -- setting initialization = "he" in the input argument. This initializes the weights to random values scaled according to a paper by He et al., 2015.

    Instructions: Please quickly read over the code below, and run it. In the next part you will implement the three initialization methods that this model() calls.

    def model(X, Y, learning_rate = 0.01, num_iterations = 15000, print_cost = True, initialization = "he"):
        """
        Implements a three-layer neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SIGMOID.
        
        Arguments:
        X -- input data, of shape (2, number of examples)
        Y -- true "label" vector (containing 0 for red dots; 1 for blue dots), of shape (1, number of examples)
        learning_rate -- learning rate for gradient descent 
        num_iterations -- number of iterations to run gradient descent
        print_cost -- if True, print the cost every 1000 iterations
        initialization -- flag to choose which initialization to use ("zeros","random" or "he")
        
        Returns:
        parameters -- parameters learnt by the model
        """
            
        grads = {}
        costs = [] # to keep track of the loss
        m = X.shape[1] # number of examples
        layers_dims = [X.shape[0], 10, 5, 1]
        
        # Initialize parameters dictionary.
        if initialization == "zeros":
            parameters = initialize_parameters_zeros(layers_dims)
        elif initialization == "random":
            parameters = initialize_parameters_random(layers_dims)
        elif initialization == "he":
            parameters = initialize_parameters_he(layers_dims)
    
        # Loop (gradient descent)
    
        for i in range(0, num_iterations):
    
            # Forward propagation: LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID.
            a3, cache = forward_propagation(X, parameters)
            
            # Loss
            cost = compute_loss(a3, Y)
    
            # Backward propagation.
            grads = backward_propagation(X, Y, cache)
            
            # Update parameters.
            parameters = update_parameters(parameters, grads, learning_rate)
            
            # Print the loss every 1000 iterations
            if print_cost and i % 1000 == 0:
                print("Cost after iteration {}: {}".format(i, cost))
                costs.append(cost)
                
        # plot the loss
        plt.plot(costs)
        plt.ylabel('cost')
        plt.xlabel('iterations (per hundreds)')
        plt.title("Learning rate =" + str(learning_rate))
        plt.show()
        
        return parameters

    2 - Zero initialization

    There are two types of parameters to initialize in a neural network:

    the weight matrices:$(W^{[1]}, W^{[2]}, W^{[3]}, ..., W^{[L-1]}, W^{[L]})$

    the bias vectors:$(b^{[1]}, b^{[2]}, b^{[3]}, ..., b^{[L-1]}, b^{[L]})$

    Exercise: Implement the following function to initialize all parameters to zeros. You'll see later that this does not work well since it fails to "break symmetry", but lets try it anyway and see what happens. Use np.zeros((..,..)) with the correct shapes.

    def initialize_parameters_zeros(layers_dims):
        """
        Arguments:
        layer_dims -- python array (list) containing the size of each layer.
        
        Returns:
        parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                        W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                        b1 -- bias vector of shape (layers_dims[1], 1)
                        ...
                        WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                        bL -- bias vector of shape (layers_dims[L], 1)
        """
        
        parameters = {}
        L = len(layers_dims)            # number of layers in the network
        
        for l in range(1, L):
            parameters['W' + str(l)] = np.zeros((layers_dims[l], layers_dims[l-1]))
            parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
                                                
        return parameters
    parameters = initialize_parameters_zeros([3,2,1])
    print("W1 = " + str(parameters["W1"]))
    print("b1 = " + str(parameters["b1"]))
    print("W2 = " + str(parameters["W2"]))
    print("b2 = " + str(parameters["b2"]))
    
    W1 = [[0. 0. 0.]
         [0. 0. 0.]]
    b1 = [[0.]
         [0.]]
    W2 = [[0. 0.]]
    b2 = [[0.]]
    
    parameters = model(train_X, train_Y, initialization = "zeros")
    print ("On the train set:")
    predictions_train = predict(train_X, train_Y, parameters)
    print ("On the test set:")
    predictions_test = predict(test_X, test_Y, parameters)
    plt.title("Model with Zeros initialization")
    axes = plt.gca()
    axes.set_xlim([-1.5,1.5])
    axes.set_ylim([-1.5,1.5])
    plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, np.squeeze(train_Y))

     

    The model is predicting 0 for every example.

    In general, initializing all the weights to zero results in the network failing to break symmetry. This means that every neuron in each layer will learn the same thing, and you might as well be training a neural network with ?[?]=1 for every layer, and the network is no more powerful than a linear classifier such as logistic regression.

    **What you should remember**: - The weights W^{[l]} should be initialized randomly to break symmetry. - It is however okay to initialize the biases b^{[l]}to zeros. Symmetry is still broken so long as W^{[l]}is initialized randomly.


    3 - Random initialization(掌握)

    To break symmetry, lets intialize the weights randomly. Following random initialization, each neuron can then proceed to learn a different function of its inputs. In this exercise, you will see what happens if the weights are intialized randomly, but to very large values.

    Exercise: Implement the following function to initialize your weights to large random values (scaled by *10) and your biases to zeros. Use np.random.randn(..,..) * 10 for weights and np.zeros((.., ..)) for biases. We are using a fixed np.random.seed(..) to make sure your "random" weights match ours, so don't worry if running several times your code gives you always the same initial values for the parameters.

    def initialize_parameters_random(layers_dims):
        """
        Arguments:
        layer_dims -- python array (list) containing the size of each layer.
        
        Returns:
        parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                        W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                        b1 -- bias vector of shape (layers_dims[1], 1)
                        ...
                        WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                        bL -- bias vector of shape (layers_dims[L], 1)
        """
        
        np.random.seed(3)               # This seed makes sure your "random" numbers will be the as ours
        parameters = {}
        L = len(layers_dims)            # integer representing the number of layers
        
        for l in range(1, L):
            parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1]) * 10
            parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
        
        return parameters
    parameters = model(train_X, train_Y, initialization = "random")
    print ("On the train set:")
    predictions_train = predict(train_X, train_Y, parameters)
    print ("On the test set:")
    predictions_test = predict(test_X, test_Y, parameters)

    **In summary**: - Initializing weights to very large random values does not work well. - Hopefully intializing with small random values does better. The important question is: how small should be these random values be? Lets find out in the next part!


    4 - He initialization(理解)

    Finally, try "He Initialization"; this is named for the first author of He et al., 2015. (If you have heard of "Xavier initialization", this is similar except Xavier initialization uses a scaling factor for the weights ?[?]W[l] of sqrt(1./layers_dims[l-1]) where He initialization would use sqrt(2./layers_dims[l-1]).)

    Exercise: Implement the following function to initialize your parameters with He initialization.

    Hint: This function is similar to the previous initialize_parameters_random(...). The only difference is that instead of multiplying np.random.randn(..,..) by 10, you will multiply it by \sqrt{\frac{2}{\text{dimension of the previous layer}}},which is what He initialization recommends for layers with a ReLU activation.

    # GRADED FUNCTION: initialize_parameters_he
    
    def initialize_parameters_he(layers_dims):
        """
        Arguments:
        layer_dims -- python array (list) containing the size of each layer.
        
        Returns:
        parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                        W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                        b1 -- bias vector of shape (layers_dims[1], 1)
                        ...
                        WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                        bL -- bias vector of shape (layers_dims[L], 1)
        """
        
        np.random.seed(3)
        parameters = {}
        L = len(layers_dims) - 1 # integer representing the number of layers
         
        for l in range(1, L + 1):
            ### START CODE HERE ### (≈ 2 lines of code)
            parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1]) * np.sqrt(2 / layers_dims[l-1])
            parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
            ### END CODE HERE ###
            
        return parameters

     

    展开全文
  • Initialization-free Distributed Algorithms for Optimal Resource Allocation with Feasibility Constraints and its Application to Economic Dispatch of Power Systems
  • 14443-3(Initialization+and+anticollision).pdf。
  • He initialization

    2020-08-16 09:35:44
    Initialization: 一个好的初始化可以: Speed up the convergence of gradient descent Increase the odds(胜算,几率) of gradient descent converging to a lower training (and generalization) error 初始化的...

    Initialization:
    一个好的初始化可以:
    Speed up the convergence of gradient descent
    Increase the odds(胜算,几率) of gradient descent converging to a lower training (and generalization) error

    初始化的shape:
    数据因当是(样本内容,样本数量)
    第一个参数W的设置因当是(样本数量,本层神经元数)
    Wx的设置为(输入层数,本层)

    He initialization

    def initialize_parameters_he(layers_dims):
        """
        Arguments:
        layer_dims -- python array (list) containing the size of each layer.
        
        Returns:
        parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                        W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
                        b1 -- bias vector of shape (layers_dims[1], 1)
                        ...
                        WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
                        bL -- bias vector of shape (layers_dims[L], 1)
        """
        
        np.random.seed(3)
        parameters = {}
        L = len(layers_dims) - 1 # integer representing the number of layers
         
        for l in range(1, L + 1):
            ### START CODE HERE ### (≈ 2 lines of code)
            parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l - 1]) * np.sqrt( 2 / layers_dims[l - 1])
            parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
            ### END CODE HERE ###
            
        return parameters
    
    展开全文
  • 信息融合英文文献Single-Point Bistatic Track Initialization Using Doppler in 3D
  • 在使用TensorRT将onnx模型转换为engine时报[TRT] Cuda initialization failure with error详细错误信息如下 [TensorRT] ERROR: CUDA initialization failure with error 222. Please check your CUDA installation:...

    起因

    在使用TensorRT将onnx模型转换为engine时报[TRT] Cuda initialization failure with error详细错误信息如下

    [TensorRT] ERROR: CUDA initialization failure with error 222. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

    问题原因

    这种错误的原因是因为NVIDIA驱动版本导致的,我用的是NVIDA官方的TensorRT镜像nvcr.io/nvidia/tensorrt:21.03-py3,在另外两台服务器上使用Driver Version: 460.84Driver Version: 460.80都能够运行。

    而出错这台服务器的NVIDIA驱动的版本是460.56

    解决办法

    NVIDIA驱动的版本升级到Driver Version: 460.84问题就解决了

    展开全文
  • 运行QQ提示initialization failure0x0000000c解决方法
  • Weight Initialization matters!!! 深度学习中的weight initialization对模型收敛速度和模型质量有重要影响!在ReLU activation function中推荐使用Xavier Initialization的变种,暂且称之为He Initialization:...

    深度学习中的weight initialization对模型收敛速度和模型质量有重要影响!



    • 在ReLU activation function中推荐使用Xavier Initialization的变种,暂且称之为He Initialization:
    import numpy as np
    W = np.random.randn(node_in, node_out) / np.sqrt(node_in / 2)
    
    • 使用Batch Normalization Layer可以有效降低深度网络对weight初始化的依赖:
    import tensorflow as tf
    # put this before nonlinear transformation
    layer = tf.contrib.layers.batch_norm(layer, center=True, scale=True,
                                         is_training=True)
    

    实验代码请参见我的Github

    背景

    深度学习模型训练的过程本质是对weight(即参数 W)进行更新,这需要每个参数有相应的初始值。有人可能会说:“参数初始化有什么难点?直接将所有weight初始化为0或者初始化为随机数!” 对一些简单的机器学习模型,或当optimization function是convex function时,这些简单的方法确实有效。然而对于深度学习而言,非线性函数被疯狂叠加,产生如本文题图所示的non-convex function,如何选择参数初始值便成为一个值得探讨的问题 --- 其本质是初始参数的选择应使得objective function便于被优化。事实上,在学术界这也是一个被actively研究的领域。

    TLDR里已经涵盖了本文的核心要点,下面在正文中,我们来深入了解一下前因后果。

    初始化为0的可行性?

    答案是不可行。 这是一道送分题 哈哈!为什么将所有W初始化为0是错误的呢?是因为如果所有的参数都是0,那么所有神经元的输出都将是相同的,那在back propagation的时候同一层内所有神经元的行为也是相同的 --- gradient相同,weight update也相同。这显然是一个不可接受的结果。

    可行的几种初始化方式

    • pre-training

    pre-training是早期训练神经网络的有效初始化方法,一个便于理解的例子是先使用greedy layerwise auto-encoder做unsupervised pre-training,然后再做fine-tuning。具体过程可以参见UFLDL的一个tutorial,因为这不是本文重点,就在这里简略的说一下:(1)pre-training阶段,将神经网络中的每一层取出,构造一个auto-encoder做训练,使得输入层和输出层保持一致。在这一过程中,参数得以更新,形成初始值(2)fine-tuning阶段,将pre-train过的每一层放回神经网络,利用pre-train阶段得到的参数初始值和训练数据对模型进行整体调整。在这一过程中,参数进一步被更新,形成最终模型。

    随着数据量的增加以及activation function (参见我的另一篇文章) 的发展,pre-training的概念已经渐渐发生变化。目前,从零开始训练神经网络时我们也很少采用auto-encoder进行pre-training,而是直奔主题做模型训练。不想从零开始训练神经网络时,我们往往选择一个已经训练好的在任务A上的模型(称为pre-trained model),将其放在任务B上做模型调整(称为fine-tuning)。

    • random initialization

    随机初始化是很多人目前经常使用的方法,然而这是有弊端的,一旦随机分布选择不当,就会导致网络优化陷入困境。下面举几个例子。

    核心代码见下方,完整代码请参见我的Github

    data = tf.constant(np.random.randn(2000, 800))
    layer_sizes = [800 - 50 * i for i in range(0,10)]
    num_layers = len(layer_sizes)
    
    fcs = []  # To store fully connected layers' output
    for i in range(0, num_layers - 1):
        X = data if i == 0 else fcs[i - 1]
        node_in = layer_sizes[i]
        node_out = layer_sizes[i + 1]
        W = tf.Variable(np.random.randn(node_in, node_out)) * 0.01 
        fc = tf.matmul(X, W)
        fc = tf.nn.tanh(fc)
        fcs.append(fc)
    

    这里我们创建了一个10层的神经网络,非线性变换为tanh,每一层的参数都是随机正态分布,均值为0,标准差为0.01。下图给出了每一层输出值分布的直方图。



    随着层数的增加,我们看到输出值迅速向0靠拢,在后几层中,几乎所有的输出值x都很接近0!回忆优化神经网络的back propagation算法,根据链式法则,gradient等于当前函数的gradient乘以后一层的gradient,这意味着输出值x是计算gradient中的乘法因子,直接导致gradient很小,使得参数难以被更新!

    让我们将初始值调大一些:

    W = tf.Variable(np.random.randn(node_in, node_out))
    

    均值仍然为0,标准差现在变为1,下图是每一层输出值分布的直方图:



    几乎所有的值集中在-1或1附近,神经元saturated了!注意到tanh在-1和1附近的gradient都接近0,这同样导致了gradient太小,参数难以被更新。

    • Xavier initialization

    Xavier initialization可以解决上面的问题!其初始化方式也并不复杂。Xavier初始化的基本思想是保持输入和输出的方差一致,这样就避免了所有输出值都趋向于0。注意,为了问题的简便,Xavier初始化的推导过程是基于线性函数的,但是它在一些非线性神经元中也很有效。让我们试一下:

    W = tf.Variable(np.random.randn(node_in, node_out)) / np.sqrt(node_in)
    


    Woohoo!输出值在很多层之后依然保持着良好的分布,这很有利于我们优化神经网络!之前谈到Xavier initialization是在线性函数上推导得出,这说明它对非线性函数并不具有普适性,所以这个例子仅仅说明它对tanh很有效,那么对于目前最常用的ReLU神经元呢(关于不同非线性神经元的比较请参考这里)?继续做一下实验:

    W = tf.Variable(np.random.randn(node_in, node_out)) / np.sqrt(node_in)
    ......
    fc = tf.nn.relu(fc)
    

    前面看起来还不错,后面的趋势却是越来越接近0。幸运的是,He initialization可以用来解决ReLU初始化的问题。

    • He initialization

    He initialization的思想是:在ReLU网络中,假定每一层有一半的神经元被激活,另一半为0,所以,要保持variance不变,只需要在Xavier的基础上再除以2:

    W = tf.Variable(np.random.randn(node_in,node_out)) / np.sqrt(node_in/2)
    ......
    fc = tf.nn.relu(fc)
    


    看起来效果非常好,推荐在ReLU网络中使用!

    Batch Normalization Layer

    Batch Normalization是一种巧妙而粗暴的方法来削弱bad initialization的影响,其基本思想是:If you want it, just make it!

    我们想要的是在非线性activation之前,输出值应该有比较好的分布(例如高斯分布),以便于back propagation时计算gradient,更新weight。Batch Normalization将输出值强行做一次Gaussian Normalization和线性变换:


    Batch Normalization中所有的操作都是平滑可导,这使得back propagation可以有效运行并学到相应的参数\gamma\beta。需要注意的一点是Batch Normalization在training和testing时行为有所差别。Training时\mu_\mathcal{B}\sigma_\mathcal{B}由当前batch计算得出;在Testing时\mu_\mathcal{B}\sigma_\mathcal{B}应使用Training时保存的均值或类似的经过处理的值,而不是由当前batch计算。

    随机初始化,无Batch Normalization:

    W = tf.Variable(np.random.randn(node_in, node_out)) * 0.01
    ......
    fc = tf.nn.relu(fc)
    

    随机初始化,有Batch Normalization:

    W = tf.Variable(np.random.randn(node_in, node_out)) * 0.01
    ......
    fc = tf.contrib.layers.batch_norm(fc, center=True, scale=True,
                                      is_training=True)
    fc = tf.nn.relu(fc)
    


    很容易看到,Batch Normalization的效果非常好,推荐使用!

    参考资料

    Xavier initialization是由Xavier Glorot et al.在2010年提出,He initialization是由Kaiming He et al.在2015年提出,Batch Normalization是由Sergey Ioffe et al.在2015年提出。

    另有知乎网友在评论中提到了一些其他相关工作:arxiv.org/abs/1511.0642arxiv.org/pdf/1702.0859

    1. Xavier Glorot et al., Understanding the Difficult of Training Deep Feedforward Neural Networks
    2. Kaiming He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classfication
    3. Sergey Ioffe et al., Batch Normalization: Accelerating Deep Network Training byReducing Internal Covariate Shift
    4. Standord CS231n


    原文: https://www.leiphone.com/news/201703/3qMp45aQtbxTdzmK.html

    展开全文
  • initialization method:初始化方式: hybrid initialization混合初始化:一种边界插值的方法,变量根据平均值或者特定值插值; 标准初始化:允许你自己设置流场初值; Compute from:选择一个zone或者all-zones,...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 233,328
精华内容 93,331
关键字:

initialization