OSPFv3 may be simpler than OSPFv2 with respect to how Authentication is handled. OSPF itslef does not have any Authentication functionality built in, instead it uses the native AH and ESP functionality built into IPv6.
To enable this under the interface you would configure ipv6 ospf authentication to enable AH or you would use ipv6 ospf encryption to enable ESP. They are mutually exclusive so you can’t have both. Only AH or ESP, usually you’d want to choose ESP because it has built in authentication as well.
Using Encryption or Authentication requires you to define a crytpgraphic algorithm and supplying keys that are used for hashing and encryption. Together, all these features form to create a security association that defines how packets should be protected by IPSEC. Usually ISAKMP/IKE is used for security association negotiations, with OSPFv3, all these parameters must be specified manually and must match on all routers that mutually authenticate themselves or encrypt OSPFv3 packets.