Input:(N,∗)where∗means any number of additional dimensions
Output:(N,∗), same shape as input
Swish(x)=xsigmoid(β1x)
ESwish(x)=βxsigmoid(x)
SILU(x)=xsigmoid(x)
Flatten T-Swish(x)={xsigmoid(x)0if x≥0otherwise
beta - βparameter used for E-Swish formulation. Default: 1.375
Input:(N,∗)where∗means any number of additional dimensions
Output:(N,∗), same shape as input
Aria2(x)=(1+e−β∗x)−α
beta -βis the exponential growth rate. Default: 0.5
alpha -αis a hyper-parameter which has a two-fold effect; it reduces the curvature in 3rd quadrant as well as increases the curvature in first quadrant while lowering the value of activation. Default: 1.0
Input:(N,∗)where∗means any number of additional dimensions
Hard ELiSH(x)={xmax(0,min(1,(x+1)/2))(ex−1)max(0,min(1,(x+1)/2))if x≥0otherwise
Input:(N,∗)where∗means any number of additional dimensions
Output:(N,∗), same shape as input
ISRU(x)=1+αx2x
ISRLU(x)={x1+αx2xif x≥0otherwise
alpha -hyperparameterαcontrols the value to which an ISRLU saturates for negative inputs. Default: 1.0
Input:(N,∗)where∗means any number of additional dimensions
Output:(N,∗), same shape as input
NLReLU(x)=ln(βmax(0,x)+1.0)
beta - βparameter used for NLReLU formulation. Default: 1.0
Input:(N,∗)where∗means any number of additional dimensions
Output:(N,∗), same shape as input
Soft Clipping(x)=α1log(1+eα(x−1)1+eαx)
alpha -αhyper-parameter, which determines how close to linear the central region is and how sharply the linear region turns to the asymptotic values. Default: 0.5
Input:(N,∗)where∗means any number of additional dimensions