fastreid.layers.activation.GELU Class Reference

Inheritance diagram for fastreid.layers.activation.GELU:

Public Member Functions
	forward (self, x)

Detailed Description

Paper Section 3.4, last paragraph notice that BERT used the GELU instead of RELU

Definition at line 53 of file activation.py.

Member Function Documentation

forward()

fastreid.layers.activation.GELU.forward	(		self,
			x )

Definition at line 58 of file activation.py.

    def forward(self, x):
        return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))

---

The documentation for this class was generated from the following file:

• smreid/fastreid/layers/activation.py