Hello, I am trying to build a neural network for accurately predicting values of a function. It takes in coordinates and gives one output - the function value.
When I asked my teacher (uni) about "rules of thumb" for choice of activation functions, network architecture & initial values of the net so I could determine what parameters would return the best performance i basically got an answer along the lines of "try different things, see what works best".
Is this really all there is to neural network design - trying different activation functions, learning rates, number of layers, etc., or are there more sophisticated ways of choosing/determining what parameters would be the best, or at least sufficiently good?