Activation Functions

This page contains the list of all activation functions supported in Echo.

Echo contains implementation for following activation functions (✅ - implemented functions, 🕑 - functions to be implemented soon, ⬜ - function is implemented in the original deep learning package):

#	Function	PyTorch	TensorFlow	MegEngine
1	Mish	✅	✅	✅
2	Swish	✅	✅	✅
3	ESwish	✅	✅	✅
4	Aria2	✅	✅	✅
5	ELiSH	✅	✅	✅
6	HardELiSH	✅	✅	✅
7	Flatten T-Swish	✅	✅	✅
8	SQNL	✅	✅	✅
9	ISRU	✅	✅	✅
10	ISRLU	✅	✅	✅
11	Soft Clipping	✅	✅	✅
12	SReLU	✅	✅	✅
13	BReLU	✅	✅	🕑
14	APL	✅	✅	🕑
15	Soft Exponential	✅	✅	✅
16	Maxout	✅	✅	🕑
17	RReLU	⬜	🕑	🕑
18	CELU	⬜	✅	🕑
19	GLU	⬜	🕑	🕑
20	SiLU	✅	✅	🕑
21	CReLU	🕑	✅	🕑
22	ProbAct	🕑	🕑	🕑
23	Noisy Activation Function	🕑	🕑	🕑
24	NLReLU	✅	✅	🕑
25	Funnel	✅	🕑	✅
26	Dynamic ReLU	🕑	🕑	🕑
27	AReLU	✅	🕑	🕑
28	SLAF	✅	🕑	🕑
29	FReLU	✅	🕑	🕑
30	Parametric Flatten T-Swish	✅	🕑	🕑
31	Adaptive Blending Units	🕑	🕑	🕑
32	ACON	🕑	🕑	🕑
35	EIS	✅	🕑	🕑
36	TanhSoft	✅	🕑	🕑
37	MPeLU	✅	🕑	🕑
38	MeLU	🕑	🕑	🕑
39	Dynamic ShiftMax	🕑	🕑	🕑
40	APLU	🕑	🕑	🕑
41	MBA	🕑	🕑	🕑
42	SAAF	🕑	🕑	🕑
43	DICE	✅	🕑	🕑
44	XUnit	✅	🕑	🕑
45	ProbAct	✅	🕑	🕑
46	Seagull	✅	🕑	🕑
47	Snake	✅	🕑	🕑
48	SIREN	✅	🕑	🕑