0%

pytorch tensorflow教程：

发表于 更新于 分类于 阅读次数：
本文字数： 2.1k 阅读时长 ≈ 2 分钟

2020年4月4日 上午9:46

tensorflow

GitHub - vahidk/EffectiveTensorflow: TensorFlow 1.x and 2.x tutorials and best practices.

pytorch[持续更新中]

GitHub - vahidk/EffectivePyTorch: PyTorch tutorials and best practices.

作者从pytorch和numpy的对比出发开始讲解：

  • The most important advantage of PyTorch over NumPy is its automatic differentiation functionality which is very useful in optimization applications such as optimizing parameters of a neural network.
  • PyTorch allows you to perform your computations on CPUs, GPUs, and TPUs without any material change to your code.
  • PyTorch also makes it easy to distribute your computation across multiple devices or machines

    Part I: PyTorch Fundamentals

  1. PyTorch basics
    1. 解释pytorch中的每轮迭代训练，更新参数W的过程
  2. Encapsulate your model with Modules
    1. In the previous example we used bare bone tensors and tensor oeprations to build our model. To make your code slightly more organized it’s recommended to use PyTorch’s modules. A module is simply a container for your parameters and encapsulates model operations.
  3. Broadcasting the good and the ugly
    1. when you have a singular dimension. PyTorch implicitly tiles the tensor across its singular dimensions to match the shape of the other operand
      1. 只有singular dimension时，才会有broadcasting
    2. when rank of two tensors don’t match, PyTorch automatically expands the first dimension of the tensor with lower rank before the elementwise operation, so the result of addition would be [[2, 3], [3, 4] ], and the reducing over all parameters would give us 12.
      1. 我理解就是从内到外的顺序
      2. A:[ [1.], [2.] ] ——> [ [1.,1.], [2.,2.] ]
      3. B:[1., 2.] -> [ [1., 2.] [1., 2.] ]
      4. A + B = [[2, 3], [3, 4] ]
        1
        2
        3
        a = torch.tensor([[1.], [2.]])
        b = torch.tensor([1., 2.])
        c = torch.sum(a + b)
  4. Take advantage of the overloaded operators
    1. 理解加速的原因？
      1. the reason is that we are calling the slice op 500 times
      2. A better choice would have been to use torch.unbind op to slice the matrix into a list of vectors all at once
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        z = -x  # z = torch.neg(x)
        z = x + y  # z = torch.add(x, y)
        z = x - y
        z = x * y  # z = torch.mul(x, y)
        z = x / y  # z = torch.div(x, y)
        z = x // y
        z = x % y
        z = x ** y  # z = torch.pow(x, y)
        z = x @ y  # z = torch.matmul(x, y)
        z = x > y
        z = x >= y
        z = x < y
        z = x <= y
        z = abs(x)  # z = torch.abs(x)
        z = x & y
        z = x | y
        z = x ^ y  # z = torch.logical_xor(x, y)
        z = ~x  # z = torch.logical_not(x)
        z = x == y  # z = torch.eq(x, y)
        z = x != y  # z = torch.ne(x, y)
  5. Optimizing runtime with TorchScript
    1. 补充学习： pytorch模型如何用C++进行加载
  6. Numerical stability in PyTorch
    1. LogSumExp这一机器学习中常见的模式-半导体新闻-摩尔芯球
      1. 这篇文章详细的讲解了logSumExp的推理过程
      2. 涉及到两个数学知识：
        1. 幂的乘法法则
        2. 对数的和差公式