# PyTorch

## Mish

```python
echoAI.Activation.t_ops.Mish()
```

Applies the element-wise function:

$$
\textbf{Mish}(x)=x\tanh(\text{softplus}(x))
$$

#### Shape: <a href="#mish-shape" id="mish-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#mish-reference" id="mish-reference"></a>

[Mish: A Self Regularized Non-Monotonic Activation Function](https://www.bmvc2020-conference.com/assets/papers/0928.pdf)

## Swish

```python
echoAI.Activation.t_ops.Swish(eswish = False, swish = True, beta = 1.735, flatten = False, pfts = False)
```

Allows the following element-wise functions:

$$
\textbf{Swish}(x)=x\text{sigmoid}(\beta\_{1} x)
$$

$$
\textbf{ESwish}(x)=\beta x\text{sigmoid}(x)
$$

$$
\textbf{SILU}(x)=x\text{sigmoid}(x)
$$

$$
\textbf{Flatten T-Swish}(x)= \begin{cases}
x\text{sigmoid}(x) + c & \text{if } x\geq 0\\
c              & \text{otherwise}
\end{cases}
$$

#### Parameters: <a href="#swish-parameters" id="swish-parameters"></a>

* **eswish** - Uses E-Swish activation function. Default: `False`.
* **swish** - Uses Swish activation function. Default: `False`.
* **flatten** - Uses Flatten T-Swish activation function. *c* is a constant of value -0.2. Default: `False`.
* **beta** - $$\beta$$parameter used for E-Swish formulation. Default: 1.375
* **pfts** - Uses Parametric Flatten T-Swish function. Has the same formulation as Flatten T-Swish with only *c* being a trainable parameter initialized with the value of -0.2 instead of setting as a constant. Default: `False`.&#x20;

{% hint style="info" %}
Note: By default, SILU is initialized.
{% endhint %}

#### Shape: <a href="#swish-shape" id="swish-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### References: <a href="#swish-reference" id="swish-reference"></a>

[Searching for Activation Functions](https://arxiv.org/abs/1710.05941)

[E-swish: Adjusting Activations to Different Network Depths](https://arxiv.org/abs/1801.07145)

[Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning](https://arxiv.org/abs/1812.06247)

[Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://arxiv.org/abs/1702.03118)

[Parametric Flatten-T Swish: An Adaptive Non-linear Activation Function For Deep Learning](https://arxiv.org/abs/2011.03155)

## Aria2

```python
echoAI.Activation.t_ops.Aria2(beta = 0.5, alpha = 1.0)
```

Applies the element-wise function:

$$
\textbf{Aria2}(x)= {(1+e^{-\beta \ast x})}^{-\alpha}
$$

#### Parameters: <a href="#aria-parameters" id="aria-parameters"></a>

* **beta** -$$\beta$$is the exponential growth rate. Default: 0.5
* **alpha** -$$\alpha$$is a hyper-parameter which has a two-fold effect; it reduces the curvature in 3rd quadrant as well as increases the curvature in first quadrant while lowering the value of activation. Default: 1.0

#### Shape: <a href="#aria-shape" id="aria-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#aria-reference" id="aria-reference"></a>

[ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets](https://arxiv.org/abs/1805.08878)

## BReLU

```python
echoAI.Activation.t_ops.BReLU()
```

Applies the element-wise function:

$$
\textbf{BReLU}(x\_{i})= \begin{cases}
f(x\_{i}) & \text{if } \text{\textit{i} mod 2 = 0}\\
-f(-x\_{i}) & \text{if } \text{\textit{i} mod 2} \neq \text{0}
\end{cases}
$$

#### Shape: <a href="#brelu-shape" id="brelu-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#brelu-reference" id="brelu-reference"></a>

[Shifting Mean Activation Towards Zero with Bipolar Activation Functions](https://arxiv.org/abs/1709.04054)

## APL

```python
echoAI.Activation.t_ops.APL(s)
```

Applies the element-wise function:

$$
\textbf{APL}(x)= \max(0,x) + \sum^{S}*{s=1}{a*{i}^{s} \ast \max(0, -x+b\_{i}^{s})}
$$

#### Parameter: <a href="#apl-parameters" id="apl-parameters"></a>

* **s** - hyperparameter, number of hinges to be set in advance. Default: 1

#### Shape: <a href="#apl-shape" id="apl-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#apl-reference" id="apl-reference"></a>

[Learning Activation Functions to Improve Deep Neural Networks](https://arxiv.org/abs/1412.6830)&#x20;

## ELiSH

```python
echoAI.Activation.t_ops.Elish(hard = False)
```

Allows the following element-wise functions:

$$
\textbf{ELiSH}(x)= \begin{cases}
x\text{sigmoid}(x) & \text{if } x \geq 0\\
(e^{x}-1)\text{sigmoid}(x) & \text{otherwise}
\end{cases}
$$

$$
\textbf{Hard ELiSH}(x)= \begin{cases}
x\max(0, \min(1,(x+1)/2)) & \text{if } x \geq 0\\
(e^{x}-1)\max(0, \min(1,(x+1)/2)) & \text{otherwise}
\end{cases}
$$

#### Parameter: <a href="#elish-parameters" id="elish-parameters"></a>

* **hard** - Uses Hard ELiSH activation function. Default: `False`

{% hint style="info" %}
Note: By default, ELiSH is initialized.
{% endhint %}

#### Shape: <a href="#elish-shape" id="elish-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#elish-references" id="elish-references"></a>

[The Quest for the Golden Activation Function](https://arxiv.org/abs/1808.00783)

## ISRU

```python
echoAI.Activation.t_ops.ISRU(alpha = 1.0, isrlu = False)
```

Allows the following element-wise functions:

$$
\textbf{ISRU}(x)= \frac{x}{\sqrt{1+\alpha x^{2}}}
$$

$$
\textbf{ISRLU}(x)= \begin{cases}
x & \text{if } x \geq 0\\
\frac{x}{\sqrt{1+\alpha x^{2}}} & \text{otherwise}
\end{cases}
$$

#### Parameters: <a href="#isru-parameters" id="isru-parameters"></a>

* **alpha** - hyperparameter$$\alpha$$controls the value to which an ISRLU saturates for negative inputs. Default: 1.0
* **isrlu** - Uses ISRLU activation function. Default: `False`

{% hint style="info" %}
Note: By default, ISRU is initialized.
{% endhint %}

#### Shape: <a href="#isru-shape" id="isru-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#isru-references" id="isru-references"></a>

[Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)](https://arxiv.org/abs/1710.09967)

## Maxout

```python
echoAI.Activation.t_ops.Maxout()
```

Applies the element-wise function:

$$
\textbf{Maxout}(\vec{x})= \max\_{i}(x\_{i})
$$

#### Shape: <a href="#maxout-shape" id="maxout-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#maxout-reference" id="maxout-reference"></a>

[Maxout Networks](https://arxiv.org/abs/1302.4389)

## NLReLU

```python
echoAI.Activation.t_ops.NLReLU(beta = 1.0, inplace = False)
```

Applies the element-wise function:

$$
\textbf{NLReLU}(x)= \ln(\beta\max(0,x)+1.0)
$$

#### Parameters: <a href="#nlrelu-shape" id="nlrelu-shape"></a>

* **beta** - $$\beta$$parameter used for NLReLU formulation. Default: 1.0
* **inplace** - can optionally do the operation in-place. Default: `False`

#### Shape: <a href="#nlrelu-shape" id="nlrelu-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#nlrelu-reference" id="nlrelu-reference"></a>

[Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks](https://arxiv.org/abs/1908.03682)

## Soft Clipping

```python
echoAI.Activation.t_ops.SoftClipping(alpha = 0.5)
```

Applies the element-wise function:

$$
\textbf{Soft Clipping}(x)= \frac{1}{\alpha}\log{\big(\frac{1+e^{\alpha x}}{1+e^{\alpha (x-1)}}\big)}
$$

#### Parameter: <a href="#softclipping-parameters" id="softclipping-parameters"></a>

* **alpha** -$$\alpha$$hyper-parameter, which determines how close to linear the central region is and how sharply the linear region turns to the asymptotic values. Default: 0.5

#### Shape: <a href="#softclipping-shape" id="softclipping-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#softclipping-reference" id="softclipping-reference"></a>

[Neural Network-Based Approach to Phase Space Integration](https://arxiv.org/abs/1810.11509)

## Soft Exponential

```python
echoAI.Activation.t_ops.SoftExponential(alpha = None)
```

Applies the element-wise function:

$$
\textbf{Soft Exponential}(x)= \begin{cases}
\frac{-\log{(1+\alpha(x + \alpha))}}{\alpha} & \text{if } \alpha < 0\\
x & \text{if } \alpha = 0\\
\frac{e^{\alpha x}-1}{\alpha} & \text{if } \alpha > 0
\end{cases}
$$

#### Parameter: <a href="#softexponential-parameters" id="softexponential-parameters"></a>

* **alpha** -$$\alpha$$trainable hyper-parameter which is initialized to zero by default. Default: `None`

#### Shape: <a href="#softexponential-shape" id="softexponential-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#softexponential-reference" id="softexponential-reference"></a>

[A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks](https://arxiv.org/abs/1602.01321)

## SQNL

```python
echoAI.Activation.t_ops.SQNL()
```

Applies the element-wise function:

$$
\textbf{SQNL}(x)= \begin{cases}
1 & \text{if } x > 2\\
x - \frac{x^2}{4} & \text{if } 0 \leq x \leq 2\\
x + \frac{x^2}{4} & \text{if } -2 \leq x < 0\\
-1 & \text{if } x < -2
\end{cases}
$$

#### Shape: <a href="#sqnl-shape" id="sqnl-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#sqnl-reference" id="sqnl-reference"></a>

[SQNL: A New Computationally Efficient Activation Function](https://ieeexplore.ieee.org/document/8489043)

## SReLU

```python
echoAI.Activation.t_ops.SReLU(in_features, parameters = None)
```

Applies the element-wise function:

$$
\textbf{SReLU}(x\_{i})= \begin{cases}
t\_i^r + a\_i^r(x\_i - t\_i^r) & \text{if } x\_i \geq t\_i^r\\
x\_i & \text{if } t\_i^r > x\_i > t\_i^l\\
t\_i^l + a\_i^l(x\_i - t\_i^l) & x\_i \leq  t\_i^l
\end{cases}
$$

#### Parameters: <a href="#srelu-parameters" id="srelu-parameters"></a>

* **in\_features** - Shape of the input. Datatype: `Tuple`
* **parameters** - ( $$t^r,t^l,a^r,a^l$$ ) parameters for manual initialization, Default: `None`. If `None` is passed, parameters are initialized randomly.

#### Shape: <a href="#srelu-shape" id="srelu-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#srelu-reference" id="srelu-reference"></a>

[Deep Learning with S-shaped Rectified Linear Activation Units](https://arxiv.org/abs/1512.07030)

## Funnel

```python
echoAI.Activation.t_ops.Funnel(in_channels)
```

Applies the element-wise function:

$$
\textbf{Funnel}(x)= \max(x,\mathbb{T}(x))
$$

#### Parameter: <a href="#frelu-parameter" id="frelu-parameter"></a>

* **in\_channels** - Number of channels in the input tensor. Datatype: `Integer`

#### Shape: <a href="#frelu-shape" id="frelu-shape"></a>

* Input:$$(\mathbf{N}, \mathbf{C}, \mathbf{H}, \mathbf{W})$$where$$\mathbf{C}$$indicates the number of channels.
* Output:$$(\mathbf{N}, \mathbf{C}, \mathbf{H}, \mathbf{W})$$, same shape as input

#### Reference: <a href="#frelu-reference" id="frelu-reference"></a>

[Funnel Activation for Visual Recognition](https://arxiv.org/abs/2007.11824)

## SLAF

```python
echoAI.Activation.t_ops.SLAF(k = 2)
```

Applies the element-wise function:

$$
\textbf{SLAF}(x)= a\_0 + a\_1 x + a\_2 x^2 + .... + a\_{N-1}x^{N-1}
$$

#### Parameter: <a href="#slaf-parameter" id="slaf-parameter"></a>

* **k** - Number of Taylor coefficients. Default: 2

#### Shape: <a href="#slaf-shape" id="slaf-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#slaf-reference" id="slaf-reference"></a>

[Learning Activation Functions: A new paradigm for understanding Neural Networks](https://arxiv.org/abs/1906.09529)

## AReLU

```python
echoAI.Activation.t_ops.AReLU(alpha = 0.90, beta = 2.0)
```

Applies the element-wise function:

$$
\textbf{AReLU}(x\_i)= \begin{cases}
C(\alpha)x\_i & \text{if } x\_i <0\\
(1+\sigma(\beta))x\_i & \text{otherwise}
\end{cases}
$$

#### Parameters: <a href="#arelu-parameters" id="arelu-parameters"></a>

* **alpha** -$$\alpha$$trainable hyper-parameter. Default: 0.90
* **beta** -$$\beta$$trainable hyper-parameter. Default: 2.0

#### Shape: <a href="#arelu-shape" id="arelu-shape"></a>

* Input:$$(\mathbf{N}, \mathbf{C}, \mathbf{H}, \mathbf{W})$$where$$\mathbf{C}$$indicates the number of channels.
* Output:$$(\mathbf{N}, \mathbf{C}, \mathbf{H}, \mathbf{W})$$, same shape as input

#### Reference: <a href="#arelu-reference" id="arelu-reference"></a>

[AReLU: Attention-based Rectified Linear Unit](https://arxiv.org/abs/2006.13858)

## FReLU

```python
echoAI.Activation.t_ops.FReLU()
```

Applies the element-wise function:

$$
\textbf{FReLU}(x)= \textbf{ReLU}(x) + b\_{l}
$$

#### Shape: <a href="#frelu-shape" id="frelu-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#frelu-reference" id="frelu-reference"></a>

[FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks](https://arxiv.org/abs/1706.08098)

## DICE

```python
echoAI.Activation.t_ops.DICE(emb_size, dim = 2, epsilon = 1e-8)
```

Applies the function:

$$
\textbf{DICE}(x)= \textbf{p(x)}\ast x + (1-\textbf{p(x)})\alpha x
$$

$$
\textbf{p(x)} = \frac{1}{1 + e ^ {-\frac{x - E(x)}{\sqrt{Var(x) + \epsilon}}}}
$$

#### Reference: <a href="#dice-reference" id="dice-reference"></a>

[Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/abs/1706.06978)

## Seagull

```python
echoAI.Activation.t_ops.Seagull()
```

Applies the function:

$$
\textbf{Seagull}(x)= \text{log}(1 + x^{2})
$$

#### Shape: <a href="#seagull-shape" id="seagull-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#seagull-reference" id="seagull-reference"></a>

[A Use of Even Activation Functions in Neural Networks](https://arxiv.org/abs/2011.11713)

## Snake

```python
echoAI.Activation.t_ops.Snake(in_features, alpha = None, alpha_trainable = True)
```

Applies the function:

$$
\textbf{Snake}(x)= x + \frac{1}{\alpha}\sin^{2}(\alpha x)
$$

#### Parameters: <a href="#snake-parameters" id="snake-parameters"></a>

* **in\_features** - shape of the input
* **alpha** -$$\alpha$$trainable hyper-parameter. Default: 1.0 when specified as `None`
* **alpha\_trainable** - switches $$\alpha$$to be a trainable parameter. Default: True

#### Shape: <a href="#snake-shape" id="snake-shape"></a>

* Input:$$(\mathbf{N}, \ast)$$where$$\ast$$means any number of additional dimensions
* Output:$$(\mathbf{N}, \ast)$$, same shape as input

#### Reference: <a href="#snake-reference" id="snake-reference"></a>

[Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf)

## SIREN

```python
echoAI.Activation.t_ops.SIREN(dim_in, dim_out, w0 = 30., c = 6., is_first = False, use_bias = True, activation = None)
```

Applies the function:

$$
\textbf{SIREN}(x)= \sin(\omega\_{0}\ast\text{linear}(x))
$$

#### Parameters: <a href="#siren-parameters" id="siren-parameters"></a>

* **dim\_in** - input dimension
* **dim\_out** - output dimension
* **w0** -$$\omega\_{0}$$hyper-parameter. Default: 30.0
* **c** - hyper-parameter used in weight initialisation for the linear layer. Default: 6.
* **is\_first** - used for weight initialisation for the linear layer. Default: `False`
* **use\_bias** - initialises bias parameter for the linear layer. Default: `True`
* **activation** - used to initialise an activation function. Default: `None`. Initialises the `sine` activation function when `None`

#### Shape: <a href="#frelu-shape" id="frelu-shape"></a>

* Input:$$(\mathbf{x}, dim\_in)$$
* Output:$$(\mathbf{x}, dim\_out)$$

#### Reference: <a href="#siren-reference" id="siren-reference"></a>

[Implicit Neural Representations with Periodic Activation Functions](https://arxiv.org/abs/2006.09661)
