Vyom Pathak

I am currently working as a full-time applied scientist at Chronograph in New York, and as a part-time research assistant at the Data Science Research Lab at the University of Florida. At Chronograph, I am working on applied NLP problems in financial domain by building large foundation models. At UF, I am working on improving faithfulness in multimodal reasoning via large language models using human feedback for the DARPA funded ECOLE project. I am working under the guidance of Dr. Daisy Wang from UF, Dr. Eric Xing from CMU, and Dr. Zhiting Hu from UCSD.

I graduated from University of Florida in May 2023 with a master's degree in computer science. At UF, I worked at the junction of deep learning, and natural language processing on large scale data at the Data Science Research Lab under the guidance of Dr. Daisy Wang as a research assistant.

Last year, I interned at Amazon as an applied scientist in the Alexa Smart Home team under the guidance of Sven Eberhardt and Zeya Chen on developing novel transformer architecture for sequence-to-sequence problems. Before that, I interned at Apple as a machine learning researcher in the Siri Text-to-Speech team under the guidance of Kishore Prahallad and Eoin Murphy, where I worked on developing end-to-end acoustic model on large scale speech data.

Before master's, I interned at the Indian Space Research Organization (ISRO) as a machine learning researcher under the guidance of Dr. Arvind Sahay for developing neural network based solution for oceanographic problems. I received my bachelor's degree from Dharmsinh Desai University in computer engineering in 2021. At DDU, under the guidance of Dr. Brijesh Bhatt, I worked on automatic speech recognition system for low-resource language (Gujarati).

I am very much open to different research domains as well as cool/hard machine learning problems. You can contact me through any of the following mediums for collaborations.

Email  /  CV  /  LinkedIn  /  Github  /  Google Scholar  /  Kaggle  /  Medium

Seeking full-time Research Engineer opportunities!

Much of my research is at the intersection of speech related tasks, finetuning large language models, and the application of deep learning in various domains. Sparingly, I have worked on speech enhancement, automatic speech recognition, natural language understanding, prompting large language models, multimodal reasoning, dialogue managers, text-to-speech, conversational agents, and time series modeling.

We build upon our previous work and develop higher-order character-level and word-level language models for better constraining the beam search decoding. We also provide an ablation on how to choose a good size of the language model for beam search decoding.

We introduced a modified neural network method of deriving Inherent Optical Properties by using the remote sensing reflectance wavelengths. The proposed outperforms all other previous methods in the retrieval of each IOPs for the Insitu dataset.

We invented a two-tier post-processing technique to correct errors in an end-to-end ASR system for low-resource language (Gujarati). Firstly, we develop a novel multi-level (character and word level) language model for constraining the beam search decoding on the ASR output. On top of that, we developed a novel Spell Correction model based on multi-lingual BERT and a word-level language model. Finally, we provide a new and extensive analysis method to understand the errors made by the ASR system for the Gujarati Language.

Along the journey of understanding machine learning, I have worked on several projects ranging from large language modeling, chat bots, protein structure degradation, kaggle competitions, and digital image processing.

Language models are essential components of natural language processing that can capture general representations of language from large-scale text corpora. In recent years, various language models have been proposed, ranging from shallow word embeddings to deep contextual encoders, with different pre-training tasks, training frameworks, adaptation methods, and evaluation benchmarks. The year 2022 saw a surge in the development of large generative models, referred to as "foundation models," that can perform multiple tasks by training on a general unlabeled dataset. However, there is a need for a comprehensive survey that can link these models and contrast their advantages and disadvantages. In this paper, we present a systematic survey of foundation language models that aims to connect various models based on multiple criteria, such as representation learning, model size, task capabilities, research questions, and practical task capabilities.

We compared 2 methods to predict the degradation of the mRNA Vaccine protein degradation. We investigate graph-based as well as sequence-to-sequence models for the RNA structure on the Eterna Dataset. Our results show that the graph-based solution better represents the dataset thus giving better performance than the sequence-to-sequence model.

We present an ensemble of image-only convolutional neural network (CNN) models with different backbones and input sizes along with a self-supervised model to classify skin lesions. We use the Bootstrap your own latent (BYOL) model for self-supervision on the state-of-the-art image recognition models for performance improvements. With this, we show an improvement of 2% over the baselines.

We have developed a Schema-based Dialogue System for Zero-Shot Task Transfer which enables designers to quickly and easily transfer the learned behaviors from related tasks/domains to new domains or tasks. We used zero-shot prompt-based Dialogue-GPT2 for the response generation part. We verified the viability of the system as a wire-framing technique through two user studies.

We participated in a Kaggle competition to predict the difficulty of a comprehensive paragraph as a regression score. We developed a novel 2D-Attention mechanism to boost our performance, along with finetuning several large language models. The final model was an ensemble based on the Forward OOF method which helped us achieve a Silver medal in the competition.

For the 'Build for Digital India' program, we decided to build a guide-bot for the Regional Transport Office (RTO), Government of India under the theme Smart cities & Infrastructure. A guide-bot that can answer and help people with relevant kinds of queries and confusions usually they have. So the work of people will become easier and correct information can be provided.

A repository for ML algorithms I implemented ranging from regression models, decision trees, CNN, and NLP concepts; to transfer learning. This repository also houses different papers that I have reviewed along with my thoughts on the same. I update this project with my new learnings.

This application is used for editing photos. It uses digital image processing based on matrix manipulations for photo editing. needs. The software provides functionalities like image RGB value modification, image styling, image flipping along the axis, image rotation as well as applying some filters on the images like Sepia, Greyscaling, Inverting, etc.

I write about my research and other interesting things including my professional journey, and other helpful guides on the medium.

We published three blogs about understanding what is Speech Recognition, feature extraction for speech recognition, to approaches to solving the problem using machine learning in the respective blog posts.

This is a rough guide on starting one’s journey in machine learning, from foundational statistics to advanced deep learning concepts.

My story of landing machine learning internships at Apple and Amazon. A detailed outlook on my professional journey from June 2017 to February 2022.

I have made contributions in the form of model implementation, bug fixes, document fixes, type fixing, function deprecation, and adding a testing suite. Some of my contributions are in the following open-source projects: PyTorch Ignite, Hugging Face, TensorFlow, Keras, Scikit-learn, Optuna, Pandas, Julia, and Python. You can find more information on my github README.

*These authors contributed equally to this work.

Big thank you to Jon Barron for the website template! Last updated: 1^st Sept 2023