VED PIYUSH, PhD

PhD, Statistics  /  University of Nebraska–LincolnVed Piyush

My research asks how a prediction should carry honest uncertainty — and what it takes to earn it.

Education

Statistics · 2011–2023
Aug 2018 – Dec 2023 PhD, Statistics University of Nebraska–Lincoln · GPA 3.99 Dissertation: A Generalized Stacking Method Using Matrix Ensemble Kalman Filter-Based Multi-Arm Neural Network. Advisor: Dr. Souparno Ghosh.
Aug 2018 – May 2020 MS, Statistics University of Nebraska–Lincoln · GPA 4.00 Earned en route to the doctorate.
Sep 2015 – Jul 2018 MS, Statistics University of Minnesota–Twin Cities · GPA 3.76 Advisor: Dr. Kean Ming Tan. Thesis work on interpretable recommendation for implicit-feedback data.
Jul 2011 – May 2014 BSc (Honours), Statistics Sri Venkateswara College, University of Delhi First Division.

Research programme

Methodology → application
Forecasting

Every forecast ships with its error bars

A weighted ensemble of ARIMA, ETS, regression and tree-based learners predicting IBM's global revenue across business units — delivering intervals, not just point estimates. Built on statsmodels, pmdarima, statsforecast and scikit-learn, with ensemble weights optimised per unit against held-out validation.

$62B revenue under forecast
Doctoral research

A Kalman filter that stacks deep learners

My dissertation: a Matrix Ensemble Kalman Filter multi-arm neural network that combines several deep learners in situ — matching transfer-learning accuracy while producing tighter prediction intervals at higher coverage than Monte Carlo Dropout. Applied to cancer drug-response prediction with Argonne National Laboratory.

PhD, Statistics · Nebraska 2023
Computational biology

Reading genomes the way we read language

Gut bacteria digest complex sugars using clusters of genes that act together. Which sugar a given cluster targets normally has to be established in the lab. Treating a cluster as a sentence of gene “words” lets representations be learned from abundant unlabelled sequence, so the few laboratory-confirmed examples are spent on the classifier rather than on learning the vocabulary.

Nucleic Acids Research · Sensors · Genetics
Retrieval-augmented generation

Eight thousand people, one grounded answer

An enterprise RAG assistant built end to end — data scoping, web scraping, ingestion into Milvus and FAISS, and a benchmark of open-source against proprietary foundation models on retrieval quality, latency and cost-performance.

8,000+ IBM employees served
Agentic AI

Agents that hand work to each other

A multi-agent orchestrator for IBM's marketing teams. It ingests raw event material — prospectuses, briefs, supporting assets — and coordinates tool calls across a multi-turn pipeline of upload, multi-format extraction, embedding and retrieval-augmented generation, so the copy it produces stays grounded in the source documents rather than invented.

Deployed for IBM marketing teams

Industry experience

Applied statistics in practice · 2016–2026

Nearly three years on staff at IBM's Chief Analytics Office, preceded by research internships at LinkedIn, Seagate, Travelers and Hennepin County. Each was a chance to test statistical methodology against a problem someone actually needed solved, on a deadline, with real data.

$3M
manual review effort
eliminated
8,000+
employees served by
the RAG assistant
$62B
global revenue under
ensemble forecast
3
industry awards
in 2025

Senior Data Scientist

IBM · Chief Analytics Office

Sep 2023 – Apr 2026Manhattan, New York

Statistical and machine-learning methodology applied across the company, from classical time-series forecasting through to large language models and multi-agent systems. Four systems reached production:

  • DocEval — GenAI document validation. Led a generative-AI pipeline that automated compliance review of thousands of business-partner submissions against predefined criteria, saving roughly $3M in manual effort. Built the multi-format extraction layer (PDF, DOCX, PNG, JPG, EML, MSG) and ran the model tradeoff study across context-window size, latency, cost and accuracy.
  • Enterprise RAG assistant. Designed and deployed a retrieval-augmented assistant serving 8,000+ employees end to end — data scoping, web scraping, ingestion into Milvus and FAISS, and a benchmark of open-source against proprietary foundation models on retrieval quality, latency and cost.
  • Agentic content platform. A multi-agent orchestrator for marketing teams that coordinates tool calls across extraction, embedding and grounded generation, turning raw event material into publish-ready copy.
  • Global revenue forecasting. A weighted ensemble of ARIMA, ETS, regression and tree-based learners forecasting revenue across business units, with weights optimised per unit against held-out validation.

Senior Data Science Intern

IBM · Chief Analytics Office

May – Aug 2022Armonk, New York

Built machine-learning models recommending strategic products to IBM clients, paired with an interpretability framework that explained each recommendation in terms of competitive parity — so a seller could say not just what to offer but why it beat the alternative.

Applied Research Data Science Intern

LinkedIn · Premium Data Science Group

May – Aug 2021Silicon Valley, California

Two pieces of work in the Premium subscriptions group:

  • Built multi-input deep-learning models for subscription-cancellation prediction, combining several distinct signal types about a member into one churn model.
  • Designed an optimised A/B testing framework to shorten the measurement cycle and accelerate the software development lifecycle.

Data Science Intern

Seagate Technology · Global Wafer Systems

Jan – Jul 2018Bloomington, Minnesota

Deployed a recommendation system that surfaced internal analytics reports to employees across two global sites, and built the surrounding machinery to keep it honest:

  • Automated the full pipeline — extraction, cleaning, model building, and writing recommendations to the production database.
  • Identified the metrics by which recommendation quality would be judged, and built an R Shiny dashboard to monitor accuracy and relevancy against them.
  • Designed tests comparing competing recommendation algorithms and evaluation strategies, and worked with web developers on how recommendations should be presented.

Energy Data Analysis Intern

Hennepin County · Facility Services

Oct – Dec 2017Minneapolis, Minnesota

Built statistical models quantifying the reduction in expenditure attributable to energy-saving practices in county-owned buildings — producing evidence-based justification for the money spent on them — and presented the findings to the energy managers of Hennepin County and the City of Minneapolis.

Advanced Analytics Intern

Travelers Insurance · Personal Insurance R&D

Jun – Aug 2016St. Paul, Minnesota

Developed models predicting the occurrence and severity of fire claims on home insurance policies, and engineered a substitute for an expensive third-party data attribute using inexpensive data already in hand — which gave the company renegotiation leverage with the vendor. Presented the methodology and findings to both technical and non-technical stakeholders.

Academic appointments

Research & teaching assistantships · 2016–2023

Five continuous years of graduate appointments at Nebraska, spanning research and instruction, preceded by three teaching assistantships at Minnesota.

Graduate Research Assistant

University of Nebraska–Lincoln & Argonne National Laboratory

Jan 2022 – Aug 2023Lincoln, Nebraska

Two research lines, running in parallel:

  • AI for precision medicine. Developed an in-situ model-averaging procedure for cancer drug-response prediction, in collaboration with researchers at Argonne National Laboratory. It matches state-of-the-art transfer-learning models on accuracy while producing prediction intervals with higher coverage and tighter width than Monte Carlo Dropout.
  • Personalised nutrition and microbiome modelling. In a bioinformatics lab — sequence embeddings over a large unsupervised microbiome corpus, then attention-based recurrent architectures for interpretable substrate classification under limited supervision.

Graduate Teaching Assistant & Instructor of Record

University of Nebraska–Lincoln · Department of Statistics

Aug 2018 – Aug 2023Lincoln, Nebraska

Held continuously across five years, covering successive teaching, instructional and research roles:

  • Instructor of record for STAT 218, Introduction to Statistics, across three terms — sole responsibility for syllabus, delivery, assessment and grading.
  • Teaching assistant and laboratory instructor for STAT 802 (Design and Analysis of Research Studies), STAT 801A (Statistical Methods in Research) and STAT 463 (Statistical Inference).
  • Demonstrated the theory and real-world application of design of experiments and statistical methods to roughly 320 students at Nebraska.

Graduate Teaching Assistant

University of Minnesota–Twin Cities · School of Statistics

2016 – 2017Minneapolis, Minnesota

Three teaching assistantships covering laboratory and discussion sessions, instructional support, grading of assignments and examinations, and office hours — for Statistical Analysis, Applied Statistics I, and Sampling Methodology in Finite Populations — reaching roughly 90 graduate and undergraduate students.

Peer-reviewed journal articles

Doctoral dissertation

Preprints & manuscripts in preparation

—

A Multi-arm ANN to combine information from multiple deep learners and attach uncertainty

V Piyush, G D Fernando, R Zhang, M R Weil, R Pal, S Ghosh

In preparation

Book chapter

In press

Introduction to Image Data Analysis

V Piyush, L Ellingson, S VanderPlas, S Ghosh

Springer · Festschrift in honour of Prof. Tathagata Bandyopadhyay

Conference proceedings

Further refereed presentations at WNAR 2023, the International Indian Statistical Association Conference 2023 and ISMB 2022 are listed on Google Scholar.

Conference presentations

Refereed · 2022–2023

Four refereed presentations at the field's recognised meetings — two statistics societies, the flagship computational-biology conference, and the plant-phenotyping network.

WNAR 2023 Western North American Region of the International Biometric Society Biometric Society regional meeting

Approximating multi-arm deep learners with in-situ trained ensembles of shallow artificial neural networks

V Piyush, G Fernando, S Ghosh

Presented the in-situ ensemble approximation: shallow networks trained alongside a deep multi-arm learner can stand in for it while carrying calibrated uncertainty — the practical core of the stacking method.

IISA 2023 International Indian Statistical Association Conference International statistics society

A Matrix Ensemble Kalman Filter-based multi-arm neural network

V Piyush, Y Yan, Y Zhou, Y Yin, S Ghosh

The methodological talk on the dissertation itself — treating stacking weights as an unobserved state updated by an ensemble Kalman filter, and what that buys in interval coverage and width.

ISMB 2022 Conference on Intelligent Systems for Molecular Biology International Society for Computational Biology

Sequence predictability score to boost classification of protein sequences

V Piyush, R Zhang, R Pal, S Ghosh

Introduced a predictability score over protein sequences and showed it improves downstream classification. ISMB is the flagship annual meeting of the International Society for Computational Biology.

NAPPN 2022 North American Plant Phenotyping Annual Conference Plant phenotyping network

Deep Learning Methods for Tassel Count Time-Series

G Fernando, V Piyush, S Ghosh

Argued that field images are not independent observations but unequally-spaced sequences carrying information about the growth trajectory — and built a hybrid model that exploits that structure for tassel counting. Archived on the ESS Open Archive.

Research threads

Question → approach → finding

Each thread is backed by a public repository, with notebook outputs committed so figures and result tables read without running anything.

I
Doctoral research · 2018–2023

Stacking deep learners so predictions carry honest uncertainty

Question
Several deep learners are available for one prediction problem. How should they be combined so the result carries calibrated uncertainty — not just a point estimate?
Approach
Treat the stacking weights as an unobserved state and update them with a Matrix Ensemble Kalman Filter, so the spread of the filter ensemble estimates predictive uncertainty directly.
Finding
Matches transfer-learning accuracy while producing prediction intervals with higher coverage and tighter width than Monte Carlo Dropout — checked in simulation where the true answer is known by construction, on classifying which sugar a bacterial gene cluster digests, and on predicting several correlated chemical properties of a molecule at once.
OutcomePreprint arXiv:2307.10436; two manuscripts in preparation; dissertation chapters 2–4.
II
With Argonne National Laboratory · 2022–2023

Calibrated prediction for cancer drug response

Question
Averaging works well when the models being averaged resemble each other. Does the uncertainty stay honest when they are built differently?
Approach
Two separately published deep-learning models predict how strongly a drug inhibits a tumour cell line, reading the same biology but organised differently. Both were brought into a shared framework so they could feed one stacker, then swept across ensemble size, prior variance, and which molecular measurements the models were given — mutations, gene expression, DNA methylation.
Finding
Interval quality holds as the models diverge. Cancer drug response is the right place to test it: the models are expensive to train, the biology is genuinely uncertain, and a bare number with no error bar is not something a clinician can act on.
III
With UNL Food Science · 2021–2023

Interpretable sequence models for microbiome data

Question
Polysaccharide utilisation loci are long gene-token sequences, labelled examples are scarce, and a black-box call is of little use to a biologist who needs to know why.
Approach
Learn representations from abundant unlabelled sequence with Word2Vec, Doc2Vec and FastText, so the scarce laboratory-confirmed labels are spent on the classifier. Two complementary routes: attention-based models benchmarked across tokenisation choices, and semi-supervised and metric-learning approaches built for the same label shortage.
Finding
Usable predictions from very little supervision, and — critically — inspectable ones: the models expose which genes drove each call, so a prediction becomes a hypothesis a microbiologist can take to the bench. Applied to real communities, including the Unified Human Gastrointestinal Genome collection and a cow rumen dataset.
Related publicationCo-author on the dbCAN-seq database update, Nucleic Acids Research 51(D1), D557–D563 — the resource this line of work builds on.
IV
Plant phenotyping · 2019–2023

Genotype, environment, and the interaction between them

Question
Yield depends on genotype, on environment, and on an interaction term — and the interaction is exactly where the difficulty lives.
Approach
A fully reproducible pipeline joining every feature type, entered as team DeepCropVision in the Genomes-to-Fields prediction competition.
Finding
Across many independent teams, diverse modelling strategies all delivered satisfactory yield estimates — published as the competition's collective result.
Related publicationsCo-author on the competition's collective result, Genetics 229(2), iyae195, and on Sensors 24(7), 2172 on maize tassel detection.
V
Recommendation systems · 2019–2022

Predicting from order alone

Question
Session-based recommendation has no stable long-run preference to estimate — only a short sequence with internal structure. How much can be predicted from order rather than identity?
Approach
Sequence-to-sequence models over listening sessions, combining user behaviour, acoustic track features and playback-device metadata in one architecture.
Finding
Session structure alone carries enough signal to predict skips and next plays without any persistent user model — presented as a departmental poster in 2021.
VI
Computer vision · 2021–2022

Making a captioning model auditable

Question
A captioning model produces fluent text, but can you see what it looked at for each word it generated?
Approach
A convolutional encoder paired with a recurrent decoder under a visual attention mechanism, so each generated token carries a distribution over image regions.
Finding
Attention does double duty — it improves the captions and localises the evidence behind each word, turning the decoder from a black box into something inspectable. Presented at Iowa State and in the UNL statistics seminar.
VII
Image analysis · 2020

Separating objects that refuse to separate

Question
How do you segment objects that are near-identical, touching and heavily overlapping — where colour cannot separate same-coloured neighbours and shape cannot distinguish a cluster from one large object?
Approach
Classical image processing rather than deep learning, so every step is inspectable: automatic thresholding to separate objects from background, lighting evened out beforehand, and watershed segmentation — treating the image as a landscape and letting basins fill until they meet — to split a clump. A support vector machine then classifies each separated object.
Finding
The pipeline decomposes end-to-end error into separation failure versus classification failure. It became a teaching case study precisely because its failure modes are visible without computing any metric.
VIII
Computer vision · 2020

Do region proposals match what a person would draw?

Question
Object detectors are trained to correct region proposals. But do classical proposals actually recover the boxes a human annotator would draw?
Approach
Compared selective-search proposals against human-annotated bounding boxes on the same images, then prepared the data for a bounding-box regressor.
Finding
This is a measurement question before it is a modelling one — where proposals and human boxes disagree systematically, a regressor trained to correct them inherits that bias rather than removing it.

Teaching

Two universities · seven courses · eleven terms

I have taught statistics from the first course a student ever takes through to graduate inference and experimental design — as sole instructor of record for three terms, and as a teaching assistant and laboratory instructor across eight more.

7
distinct courses
taught
11
terms in the
classroom
3
terms as sole
instructor of record
400+
students taught
across both universities

University of Nebraska–Lincoln

2018–2023, 2026
STAT 218 Introduction to Statistics Instructor of record

The practical application of statistical thinking to contemporary issues — collection and organisation of data, probability distributions, statistical inference, estimation and hypothesis testing. I was sole instructor across three terms, with full responsibility for the syllabus, delivery, assessment and grading — including taking the course fully online mid-pandemic.

Taught · 3 termsSpring 2020Summer 2020Fall 2021
STAT 802 Design and Analysis of Research Studies Graduate teaching assistant

A graduate course on the statistical characteristics of a research study — both designed experiments and studies where controlled experimentation is not feasible and its features must be mimicked. Covers power and the efficiency of competing designs, and the major design structures: blocking, nesting, multilevel models, split-plot and repeated measures.

Taught · 2 termsFall 2018Spring 2019
STAT 801A Statistical Methods in Research (non-calculus) Graduate teaching assistant

An introductory course for researchers who will not go beyond STAT 802/803/804 — statistical concepts and methodology for the descriptive, experimental and analytical study of biological and other natural phenomena, weighted toward practical application rather than theory.

Taught · 2 termsFall 2019Fall 2020
STAT 463 Introduction to Mathematical Statistics II: Statistical Inference Graduate teaching assistant

The theory course: interval and point estimation, sufficiency and completeness, Bayesian procedures, uniformly most powerful tests, the sequential probability ratio test, likelihood-ratio and goodness-of-fit tests, and elements of analysis of variance and nonparametrics.

TaughtSpring 2021
STAT 302 Mathematical Statistics and Modeling II Invited guest lecturer

Three invited guest sessions covering deep-learning foundations, convolutional networks on benchmark datasets, and pretrained transformer models — bringing modern machine learning into the mathematical statistics sequence.

TaughtMarch 2026

University of Minnesota–Twin Cities

2016–2017
STAT 5201 Sampling Methodology in Finite Populations Graduate teaching assistant

Finite-population sampling: simple random, stratified, cluster, unequal-probability and systematic designs, with ratio estimators, regression estimators and model-based estimation.

TaughtFall 2017
STAT 4051 Applied Statistics I Graduate teaching assistant

The first semester of the applied sequence for statistics majors — a broad survey of applied methods, how to recognise which kind of problem you have, how to choose a method to match it, and how to read the result correctly, taught on real data throughout.

TaughtFall 2016
STAT 5021 Statistical Analysis Graduate teaching assistant

A fast-paced four-credit course for graduate students who need statistics as a research technique: descriptive statistics, elementary probability, estimation, one- and two-sample tests, contingency tables, correlation, linear and multiple regression, and analysis of variance — all on real data, in software.

TaughtSpring 2016

Machine Learning and Deep Learning with Gene Sequences

Designed & led · March 2023

Three days for graduate students and postdocs in the UNL Department of Food Science — an audience with deep microbiology expertise and no deep learning. Built backwards from a question they already cared about (what substrate does this gene cluster act on) rather than forwards from architectures, so each method arrived only when it was needed.

Day one

Sequences as language

Why a gene cluster can be treated as a sentence, and what that buys you. Tokenisation choices and the consequences of getting k wrong.

Day two

Learning representations

Word2Vec, Doc2Vec and FastText trained on a large unsupervised corpus — so that representation learning never spends the scarce labelled examples.

Day three

Transfer learning & attention

Recurrent architectures trained by transfer learning on top of those embeddings, and reading attention weights as evidence rather than decoration.

Courses I would like to build

Uncertainty Quantification for Deep Learning — calibration, conformal prediction, deep ensembles and Kalman-filter approaches, taught through healthcare and decision-support cases. Applied Generative AI: Retrieval, Evaluation and Governance — foundation-model selection, retrieval-augmented generation, and the statistical problem of evaluating systems whose output is text.

Guest lecturing

Three invited sessions in STAT 302, Mathematical Statistics and Modeling II covering deep-learning foundations, convolutional networks on benchmark datasets and pretrained transformer models — bringing modern machine learning into the mathematical statistics sequence.

Outreach

Volunteer teacher of mathematics and statistics with Sweccha, a non-profit in India (2014–2015), working with children from underprivileged backgrounds and preparing lessons for widely varying levels of prior schooling.

Invited talks

Seven talks · nine venues · 2018–2026

Invited to speak at a national laboratory, at other universities' seminars and to domain departments outside statistics — where the job is to make a method land for an audience that does not share your vocabulary. The three-day workshop I designed is described under Teaching.

Invited talks & seminars

Dec 2024

Matrix Ensemble Kalman Filter Multi-arm Neural Network

Master's Seminar, Department of Statistics, California Polytechnic State University

Jul 2023

A Multi-arm ANN Model for Stacking Deep Learners and Attaching UncertaintyNational lab

Argonne National Laboratory

with G Fernando, R Zhang, R Pal and S Ghosh

Oct 2022

Developing Interpretable Recurrent Neural Networks with Attention using Transfer Learning for Gene Sequence Classification

Department of Food Science, University of Nebraska–Lincoln

Mar 2022

Feature Extraction via Word2Vec and Doc2Vec for Sentiment Analysis

STAT 885, Statistical Data Mining and Machine Learning, University of Nebraska–Lincoln

2021 & 2022

Automatic Image Captioning using Convolutional and Recurrent Neural Networks

Graphics Group, Iowa State University; and Department of Statistics Seminar, University of Nebraska–Lincoln

Delivered at two institutions

Oct 2021

Few-Shot Learning to Deal with Rare Classes in Multiclass Classification

Department of Statistics Seminar, University of Nebraska–Lincoln

2018

Collaborative Filtering for Implicit Feedback Datasets

Department of Statistics Seminar, University of Nebraska–Lincoln; and Syngenta Statistics and Data Science Student Seminar

Delivered at a university and in industry

Poster presentations

Apr 2021

Sequential Music Tracks Skip Prediction using Recurrent Neural Networks

Department of Statistics, University of Nebraska–Lincoln

Apr 2020

Segmentation and Classification of Overlapping Jellybeans

Department of Statistics, University of Nebraska–Lincoln

Statistical consulting

Collaboration outside statistics

Consulting is where a statistician earns their keep in a university — translating a collaborator's question into a design, and their data into a defensible answer. Mine has run from digital humanities through to enterprise deployment.

2017

SCOTUS Notes transcription project

University of Minnesota Libraries & Zooniverse

Statistical and natural-language-processing consultant on a crowd-sourced transcription of U.S. Supreme Court Justices' handwritten conference notes. I advised on and designed the consensus-aggregation methodology — how to combine many volunteer transcriptions of the same manuscript line into one defensible reading — and explained the algorithm and its edge-case behaviour to political scientists and library staff.

Ongoing

Design of experiments & applied consulting

University of Nebraska–Lincoln

Built on formal statistical-consulting training in STAT 8801 and STAT 825, and deepened by instructing STAT 802, Design and Analysis of Research Studies — the course that teaches researchers how to design a study that can actually answer their question, including when a controlled experiment is not available.

2023 – 2026

Enterprise AI systems

IBM Chief Analytics Office

Three years of translating research into production tools used internally and by business partners — the applied end of the same skill: understanding a stakeholder's real question, choosing a method that fits it, and being honest about what the answer can and cannot support.

Service & mentoring

Department & community
Oct 2021 & Fall 2022

Student panelist, STAT 810 Internship Panel

Department of Statistics, University of Nebraska–Lincoln

Invited back a second time to advise graduate students on securing and making the most of internships, and on moving between academic and industry research.

Spring 2019

Search committee member

Department of Statistics, University of Nebraska–Lincoln

Served on the search committee for a departmental Business Associate position — reviewing applications and interviewing candidates alongside faculty and staff.

2017 – 2018

Officer, Statistics in the Community

STATCOM, University of Minnesota–Twin Cities

Helped run the student-led organisation providing pro-bono statistical consulting to non-profit and community organisations across the Twin Cities.

During my doctoral studies I also mentored junior graduate students informally on machine-learning and deep-learning projects, and I am prepared to supervise across artificial intelligence, machine learning, uncertainty quantification and applied statistics.

Honours & fellowships

2015 – 2025
Oct 2025

IBM Outstanding Innovation Award

For the GenAI Document Validation System, DocEval.

Dec 2025

Brandon Hall Group Excellence Awards

Gold, Best Advance in Generative AI for Business Impact, for DocEval; Bronze, Best Advance in AI for Business Impact, for the enterprise RAG assistant.

2025

Stevie Awards

American Business Awards, August 2025, and International Business Awards, November 2025 — both for DocEval.

Mar 2019

Mutual of Omaha Data Science Competition

Third Place and Best in Show.

Dec 2015

Travelers Analytics Insurance Modeling Competition

First Place, University of Minnesota.

2018 – 2021

The Othmer Fellowship

University of Nebraska–Lincoln. USD 8,000 per year for the first three years of the doctoral programme, awarded to recruit exceptional scholars pursuing a terminal degree.

2017

Seagate Technology Fellowship

University of Minnesota–Twin Cities. USD 5,500; sole recipient from the statistics department that year.

What I'd like to
work on next.

My research is on prediction that carries honest uncertainty — calibration, prediction intervals, and the statistical machinery behind them — applied to genomics, health, and generative AI systems. I am looking for the next place to do that work, and I would welcome a conversation with anyone building in that space.