Figure 2 in this paper shows the "True posterior" of a VAE.
How could they have computed this? Some of you experts probably have a strong guess.
I thought the motivation for VAE is because the posterior (among other things) is truly difficult to compute?
I'm wondering what current approaches are towards this problem in an adversarial distributed ML context.
One paper I came across attempts to group gradients into "indicative features" that exhibit a certain distribution in a non-attack setting, and compare incoming gradients against this distribution and reject them if they appear to be significant outliers.
This approach seems like it would be difficult to make work in a scenario where a given "learner" (assuming some kind of distributed or peer-to-peer network where nodes learn collaboratively) only has access to a subset of a global dataset, and thus can't know what the expected distribution is for a gradient that was produced by training on data it has never seen.
I suppose this problem could maybe be attended to by polling peers to see if anyone else has knowledge about this kind of data?
Another approach I considered was simply rejecting proposed weight updates if the update would cause loss to go up. But again this seems like it would only really work if each node/learner has access to the same validation set that is representative of the global dataset. Would also be problematic in the sense that sometimes a slight loss increase is just a natural part of stochastic gradient descent.
In distributed ML settings, what are common approaches to defending against malicious nodes seeking to subvert the overall learning process via poisoning?
I'm doing on classifying problem using CNN with 20+ layers.
Because the data type is double between 0 and 1 and the floating-point arithmetic is different across compilers/platforms, the final classification result often changes on different environments with same test data+weights.
Example)
Computer 1:
First 4 outputs of layer 0: 4.2956220815 6.1773069574 6.2690120863 6.2555422433
First 4 outputs of layer 1: 6.1773069574 6.2690120863 6.2884041050 6.3546746538
First 4 outputs of layer 2: -0.0675196870 -0.0275491092 -0.0170720917 0.6978492065
First 4 outputs of layer 3: 3.2372829955 2.9805280654 2.2781343339 2.5922840719
First 4 outputs of layer 4: -0.1213837868 -0.1071539059 -0.0187412153 -0.0778026944
Computer 2:
First 4 outputs of layer 0: 4.2956220815 6.1773069574 6.2690120863 6.2555422433
First 4 outputs of layer 1: 6.1773069574 6.2690120863 6.2884041050 6.3546746538
First 4 outputs of layer 2: -0.0675196870 -0.0275491092 -0.0170720917 0.6978492065
First 4 outputs of layer 3: 3.3214833420 2.9399862621 3.1782110457 3.6103477371
First 4 outputs of layer 4: -0.1531310157 -0.1546919095 -0.0831776014 0.1733128951
After passing 4~5th layer, the classification result become totally different, making the model frail and unreliable.
How could I deal with this kind of problems..?
![[N] Human Hippocampal Neurogenesis Persists throughout Aging](http://b.thumbs.redditmedia.com/rAbz5rwi15EUJqqPPAWGWYUHcD9aHTd2HPM2KA__P9Y.jpg "[N] Human Hippocampal Neurogenesis Persists throughout Aging") | submitted by /u/phobrain [link] [comments] |
![[R]https://aeon.co/videos/a-neural-network-that-keeps-seeing-art-where-we-see-mundane-objects](http://a.thumbs.redditmedia.com/eUIlsGV6bvC1LHxcih_xvx5Kg5v2MhI1fe9F9iE7q18.jpg "[R]https://aeon.co/videos/a-neural-network-that-keeps-seeing-art-where-we-see-mundane-objects") | submitted by /u/finallyifoundvalidUN [link] [comments] |
![[P] Cuttlefish: A Lightweight Primitive for Adaptive Query Processing and Machine Learning](http://b.thumbs.redditmedia.com/Ba5OOfhIuz13JTC_XrL3GHaMFA2Gd4VQERa_-2FSBZE.jpg "[P] Cuttlefish: A Lightweight Primitive for Adaptive Query Processing and Machine Learning") | submitted by /u/dac22219 [link] [comments] |
Hi, We built comet.ml to allow machine learning engineers to automatically track their machine learning code, experiments, hyperparameters and results. We're officially out of beta and would love your feedback! Comet is free for public/open source projects and students. It's really easy to use and works no matter where you train your models.
I'm one of the founders so AMA!
![[D] Machine Learning Glossary | Google Developers](http://b.thumbs.redditmedia.com/q3R1BZdSSuyF6fM962rTM1ZcmaLnLoet_e9MfjSyiJs.jpg "[D] Machine Learning Glossary | Google Developers") | submitted by /u/sksq9 [link] [comments] |
![[P] Convolutional Generative Adversarial Network to Synthesize Novel Quilt Designs](http://b.thumbs.redditmedia.com/KhDeNXTMwF5aaLN3QVl4T01U0Vi14omvNnagTzoiwGA.jpg "[P] Convolutional Generative Adversarial Network to Synthesize Novel Quilt Designs") | submitted by /u/vishnubob [link] [comments] |
![[project] Check out this visualization of the Newton Raphson method created using my open source library - https://github.com/ryu577/pyray](http://b.thumbs.redditmedia.com/kLy0lq-sg2XLm_ST9jTrUlPZm0J1cpDprugkDqMgz0E.jpg "[project] Check out this visualization of the Newton Raphson method created using my open source library - https://github.com/ryu577/pyray") | submitted by /u/rohitpandey576 [link] [comments] |
![[P] Live stream of neural network learning to play Minecraft! (Behavioral cloning)](http://b.thumbs.redditmedia.com/zb5q4S8QrYVaahUQaNT-4g4JrEqdyReMfi_pxiyVGPA.jpg "[P] Live stream of neural network learning to play Minecraft! (Behavioral cloning)") | submitted by /u/MadcowD [link] [comments] |
I just started doing my Masters in AI this year after finishing a CS bachelor, and am intending to most likely continue on with a PhD in ML/DL (if I can). The problem is, now that I'm finally getting deep into studying ML/DL materials I'm finding that my math skills are really not good enough.
To give a specific example, I just started reading Machine Learning: A Probabilistic Perspective. Prior to picking this up, I also looked at the wildly praised Bishop's Pattern Recognition and the Elements of Statistical Learning, but it felt that the book I picked best matched the things I don't know about / want to learn.
But just as I'm working through the probability refresher chapter, I'm starting to feel my usual "I should know more about this" feeling. To be extra specific, just the mention of student's/laplace/gamma/beta distributions are something I never experienced in my previous studies. Or things like multi-variate distributions. Or even something like hypothesis testing is something that was almost untouched during my undergrad.
To give another example, I've been trying to catch up on the recent DL papers and most of the time I don't have trouble with the math, but last week I was reading the Wasserstein GAN paper and just a mention of Lipschitz continuity makes me feel like a first year undergrad.
There are many more things I could list, but you probably get the general idea.
To put this in more perspective, my undergrad was quite math-ish, yet there are things that clearly weren't covered, or they were and I have no intuition of them. There also aren't really any more courses I can take that teach more of the advanced stuff without actually taking classes with the math majors at a different faculty.
The reason I'm saying all of this, is that I can't really imagine myself finishing my thesis next year and going into a PhD where it might be expected of me to write a paper like the WGAN one.
I'm not sure how this compares to US, though I asked my professor at a deep learning seminar and he basically said that he knows of one university in Switzerland that teaches DL with lots of applied theory, and that otherwise there isn't really any course I can take to learn the more advanced stuff in probability / linear algebra / calculus.
For example, in my two probability courses we covered stuff like basic probability theory, and then moved on to markov chains, bits of queue theory, bits of poisson processes. But looking at the ML: A Prob Perspective book, it feels as if I had only taken the most basic probability theory. In Linear algebra, we did all the basic stuff, all the fun decompositions (SVD, QR, LU), eigenvalues, orthogonal matrices, etc., but we didn't touch any vector calculus, and I don't really have any intuition for any of the more advanced stuff, other than understanding the definitions/proofs. As for calculus, we kinda ended with Lagrange multipliers and did a bit about theory of metric spaces, but again, reading most of the papers, I feel like knowing what a Jacobian is is the only useful thing I know.
Any tips on how I should continue? What books to read? What online courses to take? What videos to watch?
![[D] Retro Contest | OpenAI](http://b.thumbs.redditmedia.com/u7D4tiSotUYoIAwXijDcHp2fRJ261WHctblCho_5Lys.jpg "[D] Retro Contest | OpenAI") | submitted by /u/sksq9 [link] [comments] |
