Hey, welcome back! Another week and another conference (NeurIPS) goes by and many-a-things have been trickling down the NLP pipe.
First things first, GPT-3 paper won a trophy:
Also, if you need to play catch-up, here’s a list of NLP-centric papers found at NeurIPS:
Blurred Secrets
This is Depix, a library for recovering passwords from pixelized screenshots. If you are using pixelization to safe-guard sensitive info, you need a new method. The library has already accrued 10K stars on GitHub 😭😭. Also, beurtschipper’s profile pic gets nominated for headshot of the week. This is awesome work by the way! (P.S., it requires that pixelized images were created with a linear box filter)
GitHub:
340 Cipher Goes Bye-Bye
If you like ciphers, (besides the NLP Cypher 😁) the zodiac killer’s 340 cipher was decrypted this week 👀. For background, the zodiac killer was a cold-blooded serial killer that rampaged California during the late 60s-early 70s and became famous for sending encrypted messages to authorities. His first cipher was decrypted early on, but the infamous 340 cipher remained a mystery all these years (51 to be exact). Until now… On December 5th, the deciphered Zodiac message was sent to the FBI by a small group of private citizens. To know how they cracked it, watch this:
You can now get huge memory savings by adding a single flag to your Lightning trainer. PyTorch Lightning now offers this feature on its library for those who wish to shard their training jobs across multiple GPUs. They include an easy to use sample for training a language model (from NVIDIAs NeMo library, which btw, you can find several notebooks on the Super Duper NLP Repo 😁) on the WikiText dataset.
If you want the tech stuff plus more on model parallelism from Lightning:
Found this great resource while perusing NeurIPS merch. It includes videos and slides are on all things math w/r/t machine learning.
- Overview video
- Introduction to Integration video slides
- Numerical Integration video slides
- Monte Carlo Integration video slides
- Normalizing Flows video slides
- Inference in Time Series video slides
- Backpropagation and Automatic Differentiation video slides
- Forward Backward Algorithm video slides
- Implicit Function Theorem video slides
- Method of Adjoints video slides
- Method of Lagrange video slides
- Stochastic Gradient Estimators video slides
A new Microsoft pretrained model MPNet, out of NeurIPS, combines the advantages of masked language modeling (aka BERT style) MLM and permuted language modeling PLM (aka XLNET style). Their GitHub also includes scripts for pretraining and downstream tasks such as SQuAD and Glue benchmark. Their blog post gives some background on the advantages and disadvantages of both training objectives and benchmarks compared to other models. (Found on HF’s model hub as well)
16 videos talks from Google w/r/t graph mining at NeurIPS 🔥🔥
They highlight applications of graph-based learning and graph algorithms for a wide range of areas such as detecting fraud and abuse, query clustering and duplication detection, image and multi-modal data analysis, privacy-respecting data mining and recommendation, and experimental design under interference.