Jyh-Jing Hwang

jyhjinghwang @ gmail . com

        I am currently a Research Scientist and Tech Lead Manager at Waymo Research. I've been at Waymo since 2020. Meanwhile, I taught Machine Learning (ESE-5410) and Computer Vision (CIS-5810) at UPenn MCIT Online from 2022 to 2024.
        In 2020, I received my Ph.D. degree in Computer and Information Science from University of Pennsylvania, advised by Prof. Jianbo Shi and Prof. Stella Yu at UC Berkeley / ICSI. During my study, I was fortunate to have the opportunities to intern at Facebook AI Research and Google Research.
        Before coming to the U.S., I received the B.S. and M.S. degrees in EE, advised by Prof. Liang-Gee Chen, from National Taiwan University and worked with Dr. Tyng-Luh Liu as a research assistant at Academia Sinica.
        My research interests are broadly in machine learning, including multimodal modeling, post-training optimization, reinforcement learning, diffusion models, sensor fusion, and image structures. My full publication list can be found at Google Scholar.

Publications

EMMA: End-to-End Multimodal Model for Autonomous Driving

Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, Mingxing Tan
Transactions on Machine Learning Research (TMLR), 2025

[ Paper ] [ Blog ] [ Forbes ] [ The Verge ]

S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation

Yichen Xie, Runsheng Xu, Tong He, Jyh-Jing Hwang, Katie Luo, Jingwei Ji, Hubert Lin, Letian Chen, Yiren Lu, Zhaoqi Leng, Dragomir Anguelov, Mingxing Tan
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025

[ Paper ]

LET-3D-AP: Longitudinal Error Tolerant 3D Average Precision for Camera-Only 3D Detection

Wei-Chih Hung, Vincent Casser, Henrik Kretzschmar, Jyh-Jing Hwang, Dragomir Anguelov
IEEE International Conference on Robotics and Automation (ICRA), 2024

[ Paper ]

CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection

Jyh-Jing Hwang, Henrik Kretzschmar, Joshua Manela, Sean Rafferty, Nicholas Armstrong-Crews, Tiffany Chen, Dragomir Anguelov
European Conference on Computer Vision (ECCV), 2022

[ Paper ]

Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers

Tsung-Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, Stella X. Yu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, Oral

[ Paper ] [ Code ] [ Webpage ]

Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning

Tsung-Wei Ke, Jyh-Jing Hwang, Stella X. Yu
International Conference on Learning Representations (ICLR), 2021

[ Paper ] [ Code ] [ Webpage ]

SegSort: Segmentation by Discriminative Sorting of Segments

Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell Collins, Tien-Ju Yang, Xiao Zhang, Liang-Chieh Chen
International Conference on Computer Vision (ICCV), 2019

[ Paper ] [ Code ] [ Webpage ]

Adversarial Structure Matching for Structured Prediction Tasks

Jyh-Jing Hwang*, Tsung-Wei Ke*, Jianbo Shi, Stella X. Yu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

[ Paper ] [ Code ] [ Webpage ]

DeeperLab: Single-Shot Image Parser

Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen
Technical report, 2019

[ Paper ]

Adaptive Affinity Fields for Semantic Segmentation

Tsung-Wei Ke*, Jyh-Jing Hwang*, Ziwei Liu, Stella X. Yu
European Conference on Computer Vision (ECCV), 2018

[ Paper ] [ Code ] [ Webpage ]

Learning Beyond Human Expertise with Generative Models for Dental Restorations

Jyh-Jing Hwang, Sergei Azernikov, Alexei A. Efros, Stella X. Yu
Technical report, 2018

[ Paper ]

Force from Motion: Decoding Physical Sensation in a First Person Video

Hyun Soo Park, Jyh-Jing Hwang, Jianbo Shi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, Oral

[ Paper ] [ Webpage ]

Egocentric Future Localization

Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, Oral

[ Paper ] [ Webpage ]

Pixel-wise Deep Learning for Contour Detection

Jyh-Jing Hwang, Tyng-Luh Liu
International Conference on Learning Representations (ICLR), 2015, Workshop

[ Paper ]

Past Experiences

Intern at Facebook AI Research

Worked with Dr. Ishan Misra and Dr. Laurens van der Maaten.

2019.05-2019.08

Intern at Google AI

Worked with Dr. Liang-Chieh Chen, Dr. Maxwell Collins, et al.
  • Researched "SegSort: Segment Sorting for Semantic Segmentation".
  • Participated in the development of DeeperLab.

2018.06-2018.09

Graduate Student Researcher at UC Berkeley / ICSI

Worked with Prof. Stella Yu. Affliated with Berkeley DeepDrive.
  • Researched "Adversarial Structure Matching for Structured Prediction Tasks".
  • Researched "Adaptive Affinity Fields for Semantic Segmentation".
  • Researched "Learning Beyond Human Expertise with Generative Models for Dental Restorations" and collaborated with Prof. Alexei A. Efros and Dr. Sergei Azernikov at Glidewell Dental Labs.
  • Researched several projects, e.g., vehicle detection in aerial imagery, timely attention for autonomous driving, etc..



2017.06-Now

Graduate Teaching Assistant at University of Pennsylvania

Assisted courses CIS 680, CIS 520, and CIS 581.

2017 Fall
2016 Fall
2015 Fall