/sci/ - Science & Math » Thread #12006097

18KiB, 652x466, aa.png

View Same Google iqdb SauceNAO

Anonymous Fri 14 Aug 01:13:25 2020 No.12006097 View Reply Original Report

Quoted By: >>12006337 >>12006403 >>12006416 >>12006455 >>12006674 >>12006697 >>12007596

A new activation function I invented. It's a quotient of polynomials

Anonymous

Anonymous Fri 14 Aug 2020 02:46:09 No.12006337 Report

Quoted By: >>12006364

>>12006097
Look boring as fuck.

Anonymous

Anonymous Fri 14 Aug 2020 02:55:07 No.12006364 Report

Quoted By: >>12006399

>>12006337
It's like no other function. Doesn't have small gradients, but can't explode like relu

Anonymous

Anonymous Fri 14 Aug 2020 03:05:12 No.12006399 Report

Quoted By: >>12006421

>>12006364
So, what is the use of it? Are you just inventing new stuff fot the sake of it or are you, say, want to improve your neural networks or something?

Sage

Sage Fri 14 Aug 2020 03:07:10 No.12006403 Report

Quoted By:

>>12006097
You didn't even invent this.

Anonymous

Anonymous Fri 14 Aug 2020 03:10:26 No.12006416 Report

Quoted By: >>12006424 >>12006427 >>12006520 >>12007690

>>12006097
Dude this just looks like a cube root graph...

Anonymous

Anonymous Fri 14 Aug 2020 03:11:49 No.12006421 Report

Quoted By: >>12006554

>>12006399
Relu didn't work for me. It went to 0 so the gradient went to 0. Tanh and sigmoid work, but they have small gradients so they train slow. This has large gradients and is resistant to infinity and nan.

Anonymous

Anonymous Fri 14 Aug 2020 03:12:51 No.12006424 Report

Quoted By:

>>12006416
cube root isn't differentiable. This is though.

Anonymous

Anonymous Fri 14 Aug 2020 03:14:22 No.12006427 Report

Quoted By:

>>12006416
This is different

Anonymous

Anonymous Fri 14 Aug 2020 03:24:57 No.12006455 Report

Quoted By:

>>12006097
Nice anon. Test it on MNIST or something. Compare it with some Relu variations.

Anonymous

Anonymous Fri 14 Aug 2020 03:52:32 No.12006520 Report

Quoted By:

>>12006416
It's similar to cube root. It's defined in terms of many operations including cube root

Anonymous

Anonymous Fri 14 Aug 2020 04:08:05 No.12006554 Report

Quoted By: >>12006556

>>12006421
There are loads of other variations of relu to relu to at least try anon... leaky, parametric, elu, selu, gelu, and more

Anonymous

Anonymous Fri 14 Aug 2020 04:09:07 No.12006556 Report

Quoted By: >>12006566

>>12006554
And you could just add gradient clipping to the large gradients

Anonymous

Anonymous Fri 14 Aug 2020 04:12:31 No.12006566 Report

Quoted By: >>12006668

>>12006556
Right. This is aesthetic though because it's differentiable and not piecewise. So the gradients might be simple.

Anonymous

Anonymous Fri 14 Aug 2020 05:15:11 No.12006668 Report

Quoted By: >>12006696

>>12006566
leaky relu gradient is about as simple as it gets. and the simplicity doesn't matter anyway since you're presumably just going to plug it into a framework that computes all the gradients for you

Anonymous

Anonymous Fri 14 Aug 2020 05:16:56 No.12006674 Report

Quoted By:

>>12006097
looks like a scaled arctan/tanh

Anonymous

Anonymous Fri 14 Aug 2020 05:28:37 No.12006696 Report

Quoted By:

>>12006668
I did forget about leaky relu. However, this new activation could be good. It keeps values near +/-1.

Anonymous

Anonymous Fri 14 Aug 2020 05:32:21 No.12006697 Report

Quoted By:

>>12006097
>invented
No, you discovered it.

Anonymous

Anonymous Fri 14 Aug 2020 14:15:59 No.12007596 Report

Quoted By: >>12008063

>>12006097
Functions that look sigmoidal but are very slightly linear at high input values act generally like a normal sigmoid with one extra very useful property: they're a bijection to the reals so they're invertible for every value of the output, which allows you to build an invertible network that acts just like a regular network as long as your other layers are invertible.

Anonymous

Anonymous Fri 14 Aug 2020 15:16:08 No.12007690 Report

Quoted By: >>12007748

>>12006416
not at all. brainlet

Anonymous

Anonymous Fri 14 Aug 2020 15:34:47 No.12007719 Report

Quoted By:

This seems absolutely pointless

Anonymous

Anonymous Fri 14 Aug 2020 15:58:51 No.12007748 Report

Quoted By:

>>12007690
ok child

Anonymous

Anonymous Fri 14 Aug 2020 17:58:42 No.12008063 Report

Quoted By: >>12008373

>>12007596
The logistic function is invertible too. There is a unique value for every input.

Anonymous

Anonymous Fri 14 Aug 2020 19:40:44 No.12008373 Report

Quoted By: >>12008486

>>12008063
But its range isn't the reals so it restricts the domain of the inverted network which is perfectly ok if you are ok restricting the inverted network's domain. The really cool thing about (I think theyre called) leaky logistic functions is that you could even stick an inverted leaky logistic function into your forward network as just another layer and it would be completely valid backward too. In fact there wouldn't be any difference between the forward or backward networks (though the order of certain layers is different, like the bias vectors might have to be added before you multiply your inverted weight matrices). And note that this doesn't work for higher order weight tensors, you can only do this with weight matrices

Anonymous

Anonymous Fri 14 Aug 2020 20:07:50 No.12008486 Report

Quoted By: >>12009046

>>12008373
Seems like it could be useful. For example you could train a classifier and generate stuff with it, like images.

Anonymous

Anonymous Fri 14 Aug 2020 22:47:06 No.12009046 Report

Quoted By: >>12009200

>>12008486
One thing I've been wanting to do for awhile but haven't got around to it is to make an autoencoder using this idea. Since the autoencoder only has to be trained for half the usual parameters it should theoretically be faster. Though the requirement that the network remains bijective might be a problem, I'm not sure.

If anyone tries it let us know!

Anonymous

Anonymous Fri 14 Aug 2020 23:46:39 No.12009200 Report

Quoted By:

>>12009046
I think it could allow for few-shot generative networks.

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches