Yes, that backend secret is not used for validating the token, but that secret key is used by the backend adapter to communicate with the keycloak API.

You can access the certs at this end-point, which can be used to validate the token.

https://<keycloak-host>:port/auth/realms/<REALM>/protocol/openid-connect/certs