---
description: This page contains the list of all activation functions supported in Echo.
---

# Activation Functions

Echo contains implementation for following activation functions (✅ - implemented functions, 🕑 - functions to be implemented soon, ⬜ - function is implemented in the original deep learning package):

|  #  | Function                                                                                           | PyTorch | TensorFlow | MegEngine |
| :-: | -------------------------------------------------------------------------------------------------- | :-----: | :--------: | :-------: |
|  1  | [Mish](https://www.bmvc2020-conference.com/assets/papers/0928.pdf)                                 |    ✅    |      ✅     |     ✅     |
|  2  | [Swish](https://arxiv.org/abs/1710.05941)                                                          |    ✅    |      ✅     |     ✅     |
|  3  | [ESwish](https://arxiv.org/abs/1801.07145)                                                         |    ✅    |      ✅     |     ✅     |
|  4  | [Aria2](https://arxiv.org/abs/1805.08878)                                                          |    ✅    |      ✅     |     ✅     |
|  5  | [ELiSH](https://arxiv.org/abs/1808.00783)                                                          |    ✅    |      ✅     |     ✅     |
|  6  | [HardELiSH](https://arxiv.org/abs/1808.00783)                                                      |    ✅    |      ✅     |     ✅     |
|  7  | [Flatten T-Swish](https://arxiv.org/abs/1812.06247)                                                |    ✅    |      ✅     |     ✅     |
|  8  | [SQNL](https://ieeexplore.ieee.org/document/8489043)                                               |    ✅    |      ✅     |     ✅     |
|  9  | [ISRU](https://arxiv.org/abs/1710.09967)                                                           |    ✅    |      ✅     |     ✅     |
|  10 | [ISRLU](https://arxiv.org/abs/1710.09967)                                                          |    ✅    |      ✅     |     ✅     |
|  11 | [Soft Clipping](https://arxiv.org/abs/1810.11509)                                                  |    ✅    |      ✅     |     ✅     |
|  12 | [SReLU](https://arxiv.org/abs/1512.07030)                                                          |    ✅    |      ✅     |     ✅     |
|  13 | [BReLU](https://arxiv.org/abs/1709.04054)                                                          |    ✅    |      ✅     |     🕑    |
|  14 | [APL](https://arxiv.org/abs/1412.6830)                                                             |    ✅    |      ✅     |     🕑    |
|  15 | [Soft ](https://arxiv.org/abs/1602.01321)[Exponential](https://arxiv.org/abs/1602.01321)           |    ✅    |      ✅     |     ✅     |
|  16 | [Maxout](https://arxiv.org/abs/1302.4389)                                                          |    ✅    |      ✅     |     🕑    |
|  17 | [RReLU](https://arxiv.org/abs/1505.00853)                                                          |    ⬜    |     🕑     |     🕑    |
|  18 | [CELU](https://arxiv.org/abs/1704.07483)                                                           |    ⬜    |      ✅     |     🕑    |
|  19 | [GLU](https://arxiv.org/abs/1612.08083)                                                            |    ⬜    |     🕑     |     🕑    |
|  20 | [SiLU](https://arxiv.org/abs/1702.03118)                                                           |    ✅    |      ✅     |     🕑    |
|  21 | [CReLU](https://arxiv.org/abs/1603.05201)                                                          |    🕑   |      ✅     |     🕑    |
|  22 | [ProbAct](https://arxiv.org/abs/1905.10761)                                                        |    🕑   |     🕑     |     🕑    |
|  23 | [Noisy Activation Function](https://arxiv.org/abs/1603.00391)                                      |    🕑   |     🕑     |     🕑    |
|  24 | [NLReLU](https://arxiv.org/abs/1908.03682)                                                         |    ✅    |      ✅     |     🕑    |
|  25 | [Funnel](https://arxiv.org/abs/2007.11824)                                                         |    ✅    |     🕑     |     ✅     |
|  26 | [Dynamic ReLU](https://arxiv.org/abs/2003.10027)                                                   |    🕑   |     🕑     |     🕑    |
|  27 | [AReLU](https://arxiv.org/abs/2006.13858)                                                          |    ✅    |     🕑     |     🕑    |
|  28 | [SLAF](https://arxiv.org/abs/1906.09529)                                                           |    ✅    |     🕑     |     🕑    |
|  29 | [FReLU](https://arxiv.org/abs/1706.08098)                                                          |    ✅    |     🕑     |     🕑    |
|  30 | [Parametric Flatten T-Swish](https://arxiv.org/abs/2011.03155)                                     |    ✅    |     🕑     |     🕑    |
|  31 | [Adaptive Blending Units](https://arxiv.org/abs/1806.10064)                                        |    🕑   |     🕑     |     🕑    |
|  32 | [ACON](https://arxiv.org/abs/2009.04759)                                                           |    🕑   |     🕑     |     🕑    |
|  35 | [EIS](https://arxiv.org/abs/2009.13501)                                                            |    ✅    |     🕑     |     🕑    |
|  36 | [TanhSoft](https://arxiv.org/abs/2009.03863)                                                       |    ✅    |     🕑     |     🕑    |
|  37 | [MPeLU](https://arxiv.org/abs/1606.00305)                                                          |    ✅    |     🕑     |     🕑    |
|  38 | [MeLU](https://arxiv.org/abs/1905.02473)                                                           |    🕑   |     🕑     |     🕑    |
|  39 | [Dynamic ShiftMax](https://arxiv.org/abs/2011.12289)                                               |    🕑   |     🕑     |     🕑    |
|  40 | [APLU](https://arxiv.org/abs/1412.6830)                                                            |    🕑   |     🕑     |     🕑    |
|  41 | [MBA](https://arxiv.org/abs/1604.00676)                                                            |    🕑   |     🕑     |     🕑    |
|  42 | [SAAF](http://proceedings.mlr.press/v54/hou17a.html)                                               |    🕑   |     🕑     |     🕑    |
|  43 | [DICE](https://arxiv.org/abs/1706.06978)                                                           |    ✅    |     🕑     |     🕑    |
|  44 | [XUnit](https://arxiv.org/abs/1711.06445)                                                          |    ✅    |     🕑     |     🕑    |
|  45 | [ProbAct](https://arxiv.org/abs/1905.10761)                                                        |    ✅    |     🕑     |     🕑    |
|  46 | [Seagull](https://arxiv.org/abs/2011.11713)                                                        |    ✅    |     🕑     |     🕑    |
|  47 | [Snake](https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf) |    ✅    |     🕑     |     🕑    |
|  48 | [SIREN](https://arxiv.org/abs/2006.09661)                                                          |    ✅    |     🕑     |     🕑    |
