"I have worked with images and videos before, but getting neural networks to work well with 3D information is surprisingly tricky," said Basile Van Hoorick, a third-year PhD student who worked with Vondrick to develop the framework that can understand occlusions as they occur. Unlike humans, an understanding of the three-dimensionality of our world does not come naturally to computers. The second leap in the project was not only to convert data from cameras into 3D seamlessly but also to reconstruct the entire configuration of the scene beyond what can be seen.

This work could expand the perception capabilities of home robots widely. In any indoor environment, things become hidden from view all the time. Hence, robots need to interpret their surroundings intelligently. The "soda can inside the refrigerator" situation is one of many examples. Still, it is easy to see how any application that uses vision will benefit if robots can draw upon their memory and object-permanence reasoning skills to keep track of both objects and humans as they move around the house.

Moving beyond the rigid body assumption

Most robots today are programmed with a series of assumptions for them to work. One is the rigid body assumption, which assumes that an object is solid and doesn't change shape. And that simplifies a lot of things. Roboticists can completely ignore the physics of the object the robot is interacting with and only have to think about the robot's motion.

Image

Shuran Song

The Columbia Artificial Intelligence and Robotics (CAIR) Lab, led by Computer Science Assistant Professor Shuran Song, has been researching robotic movement in a different way. Her research focuses on deformable, non-rigid objects--they fold, bend, and change shape. When working with deformable objects, roboticists can no longer rely on the rigid body assumption, forcing them to think about physics again. 

"In our work, we are trying to investigate how humans intuitively do things," said Shuran Song, also a Toyota Research Institute Young Faculty awardee. Instead of trying to account for every possible parameter, her team developed an algorithm that allows the robot to learn from doing, making it more generalizable and lessening the need for massive amounts of training data. It forced the group to rethink how people do an action, like hitting a target with a rope. We usually don’t think about the trajectory of the string--instead, we try to hit the object first and adjust our movements until we’re successful. "This new perspective was essential to solve this difficult problem in robotics,” Song noted.

"Previously, to achieve this level of precision, the robot needed to do the task maybe 100 to 1,000 times," said Cheng Chi, a third-year PhD student who worked with Song to develop IRP. "With our system, we can do it within ten times, which is about the same performance as a person."

The researchers noticed that there were still some limitations with the flinging motion that their robot could make. While the flinging motion is effective, it is limited by the speed of the robot arm, which means it cannot handle large items. Not to mention that it is dangerous to have a fast flinging motion around people. 

Song’s team took this research a step further and developed a new approach to manipulating them by using actively blown air. They armed their robot with an air pump and it was able to quickly unfold a large piece of cloth or open a plastic bag. The self-supervised learning framework they call DextAIRity learns to effectively perform a target task through a sequence of grasping or air-based blowing actions. Using visual feedback, the system uses a closed-loop formulation that continuously adjusts its blowing direction.

About the Study

JOURNAL: Nature Genetics

STUDY: “Multi-modal single-cell and whole-genome sequencing of small, frozen clinical specimens.”

AUTHORS: Yiping Wang (1,2*), Joy Linyue Fan (3*), Johannes C. Melms (1,4*), Amit Dipak Amin (1,4), Yohanna Georgis (5), Irving Barrera (6), Patricia Ho (1,4), Somnath Tagore (1,7), Gabriel Abril-Rodriguez(8), Siyu He (3), Yinuo Jin (3), Jana Biermann (1,2), Matan Hofree (6), Lindsay Caprio (1,4), Simon Berhe (4), Shaheer A. Khan (1,5), Brian S. Henick (1,5), Antoni Ribas (8,9), Evan Z. Macosko (6,10), Fei Chen (6,11), Alison M. Taylor (5,12), Gary K. Schwartz (1,5), Richard D. Carvajal (1,5), Elham Azizi (3,5,13,#), Benjamin Izar (1,2,4,5,7#)

1. Department of Medicine, Division of Hematology/Oncology, Vagelos College of Physicians and Surgeons, Columbia University Irving Medical Center
2. Department of Systems Biology, Program for Mathematical Genomics, Columbia University
3. Department of Biomedical Engineering, Columbia University
4. Columbia Center for Translational Immunology, Columbia University Irving Medical Center
5. Vagelos College of Physicians and Surgeons, Herbert Irving Comprehensive Cancer Center, Columbia University
6. Broad Institute of Harvard and MIT
7. Department of Systems Biology, Columbia University Irving Medical Center
8. Department of Medicine, Jonsson Comprehensive Cancer Center, University of California, Los Angeles (UCLA)
9. Parker Institute for Cancer Immunotherapy, San Francisco
10. Department of Psychiatry, Massachusetts General Hospital
11. Department of Stem Cell and Regenerative Biology, Harvard University
12. Department of Pathology and Cell Biology, Columbia University Irving Medical Center, New York, NY, USA
13. Irving Institute for Cancer Dynamics, Columbia University, New York, NY, USA.

FUNDING: Y.W. is supported by National Institutes of Health (NIH), National Institute of Allergy and Infectious Disease training grant (no. T32AI148099). B.I. is supported by the NIH, National Cancer Institute (NCI) (grant nos. K08CA222663, R37CA258829, R01CA266446, and U54CA225088), a Burroughs Wellcome Fund Career Award for Medical Scientists, a Velocity Fellows Award, the Louis V. Gerstner, Jr. Scholars Program and a Young Investigator Award by the Melanoma Research Alliance. R.D.C., E.A., and B.I. are supported by an NCI grant (no. R21CA263381) and a Columbia University Research Initiatives in Science & Engineering Award. E.A. was supported by an NCI grant (no. R00CA230195) and NSF grant (no. CBET-2144542). J.L.F. acknowledges support from the Columbia University Van C. Mow fellowship. G.A.-R. and A.R. are supported by the Parker Institute for Cancer Immunotherapy and an NIH grant (no. P01CA168585). A.M.T. is supported by the NCI (grant no. 5K22CA237733-03). This work was supported by an NIH/NCI Cancer Center Support grant (no. P30CA013696), the Molecular Pathology Shared Resource and its Tissue Bank at Columbia University, and the Flow-cytometry Core Facility supported by a grant (no. S10OD020056).

COI: B.I. has received consulting fees from Volastra Therapeutics Inc, Merck, AstraZeneca, and Janssen Pharmaceuticals and has received research funding to Columbia University from Alkermes, Arcus Biosciences, Checkmate Pharmaceuticals, Compugen, Immunocore, and Synthekine. G.A.-R. has received honoraria from consulting with Arcus Biosciences. A.R. has received honoraria from consulting with Amgen, Bristol Myers Squibb, Chugai, Genentech, Merck, Novartis, Roche, and Sanofi, is or has been a member of the scientific advisory board, and holds stock in Arcus, Compugen, CytomX, Highlight, ImaginAb, Isoplexis, Kite-Gilead, Lutris, Merus, PACT, RAPT, Synthekine, and Tango Therapeutics. A.M.T. receives research support from Ono Pharmaceuticals. B.S.H. participated in advisory boards for AstraZeneca and Ideaya. R.D.C. is a consultant for Alkermes, Bristol Myers Squibb, Castle Biosciences, Delcath, Eisai, Hengrui, Ideaya, Immunocore, InxMed, Iovance, Merck, Novartis, Oncosec, Pierre Fabre, PureTech Health, Regeneron, Sanofi Genzyme, Sorrento Therapeutics and Trisalus, serves on clinical/scientific advisory boards for Aura Biosciences, Chimeron, and Rgenix Research, and has received research funding to Columbia University from Amgen, Astellis, AstraZeneca, BioMed Valley, Bolt, Bristol Myers Squibb, Corvus, Cstone, Foghorn, Ideaya, Immatics, Immunocore, InxMed, Iovance, Merck, Mirati, Novartis, Pfizer, Plexxikon, Regeneron, and Roche/Genentech. B.I. and J.C.M. filed a patent describing the generation of high-quality single-cell genomics data from frozen tissues. The remaining authors declare no competing interests.

About the Study

Journal: Nature Nanotechnology

Title: “Selective targeting of visceral adiposity by polycation nanomedicine.”

Authors : Qianfen Wan (1)†, Baoding Huang (7,2)†, Tianyu Li (2), Yang Xiao (2), Ying He (1), Wen Du (3), Branden Z. Wang (1), Gregory F. Dakin (4), Michael Rosenbaum (3,5), Marcus D. Goncalves (6), Shuibing Chen (4), Kam W. Leong (2), Li Qiang (1)

1. Naomi Berrie Diabetes Center and Department of Pathology and Cell Biology, Columbia University
2. Department of Biomedical Engineering, Columbia University
3. Department of Medicine, Columbia University
4. Department of Surgery, Weill Cornell Medicine  
5. Department of Pediatrics, Columbia University
6. Department of Medicine, Department of Surgery, Weill Cornell Medicine
7. Department of Orthopedic Surgery, The Sixth Affiliated Hospital, Sun Yat-Sen University  and Guangdong Provincial Key Laboratory of Orthopedics and Traumatology; Guangzhou, China

This work was supported by Russell Berrie Foundation (L.Q. and Q.W.), Blavatnik SIRS funding (L.Q. and   K.W.L.), National Institutes of Health grant RO1AR073935 and the U.S. Army Medical Research grant W81XWH1910463 (K.W.L.), and The Manoogian Simone Foundation (M.D.G).

COI: A patent application is pending. All other authors declare no conflict of interest.

Journal: Biomaterials

Title: “Polycationic PAMAM ameliorates obesity-associated chronic inflammation and focal adiposity.”

Authors: Baoding Huang (1,2), Qianfen Wan (3), Tianyu Li (2), Calhoun Carmen (3), Kam W. Leong (2), and Li Qiang (3)

1. Department of Orthopaedic Surgery, The Sixth Affiliated Hospital, Sun Yat-sen University and Guangdong Provincial Key Laboratory of Orthopaedics and Traumatology, Guangzhou, China
2. Department of Biomedical Engineering, Columbia University
3. Naomi Berrie Diabetes Center and Department of Pathology and Cell Biology, Columbia University

The study was supported by the Russell Berrie Foundation (L.Q. and Q.W.), Blavatnik SIRS funding (L.Q. and   K.W.L.), National Institutes of Health grant RO1AR073935 and the U.S. Army Medical Research grant W81XWH1910463 (K.W.L.).

COI: The authors have patents pending. No other competing interests are declared.

About The Study

JOURNAL: Nature

TITLE: “All-optical frequency division on-chip using a single laser”

AUTHORS: Yun Zhao (APAM postdoc, former EE PhD), Jae K. Jang (former APAM postdoc), Garrett J. Beals (APAM PhD), Karl J. McNulty (EE PhD), Xingchen Ji (former EE postdoc), Yoshitomo Okawachi (former APAM research scientist), Michal Lipson (professor, EE, APAM), Alexander L. Gaeta (Professor, APAM, EE)

FUNDING: This work was supported by Defense Advanced Research Projects Agency of the U.S. Department of Defense (Grant No. HR0011-22-2-0007 ), Army Research Office (ARO) (Grant No. W911NF-21-1-0286), and Air Force Office of Scientific Research (AFOSR) (Grant No. FA9550-20-1-0297).

The authors declare no competing interests.

“Understanding whether tissues are staying healthy and getting good blood supply during surgical procedures is really important,” says Hillman. “We also realized that if we don’t have to remove (and kill) tissues to look at them, we can find many more uses for MediSCAPE, even to answer simple questions such as ‘what tissue is this?’ or to navigate around precious nerves. Both of these applications are really important for robotic and laparoscopic surgeries where surgeons are more limited in their ability to identify and interact with tissues directly.”

A critical final step for the team was to reduce the large format of the standard SCAPE microscopes in Hillman’s lab to something that would fit into an operating room and could be used by a surgeon in the human body. Post-doctoral fellow Wenxuan Liang worked with the team to develop a smaller version of the system with a better form factor, and a sterile imaging cap. PhD candidate Malte Casper helped to acquire the team’s first demonstration of MediSCAPE in a living human, collecting images of a range of tissues in and around the mouth. These results included rapidly imaging while a volunteer literally licked the end of the imaging probe, producing detailed 3D views of the papillae of the tongue.

Eager to take this technology to the next level with a larger clinical trial, the team is currently working on commercialization and FDA approval. Hillman adds, “We are just so amazed to see what MediSCAPE reveals every time we use it on a new tissue, and especially that we barely ever even needed to add dyes or stains to see structures that pathologists can recognize.”

Hillman and her team hope that MediSCAPE will make standard histology a thing of the past, putting the power of real-time histology and decision making into the surgeon’s hands.

About the Study

Journal: Nature Biomedical Engineering

Title: High-speed light-sheet microscopy for the in-situ acquisition of volumetric histological images of living tissue

Authors: Kripa B. Patel 1, Wenxuan Liang 1, Malte J. Casper 1, Venkatakaushik Voleti1, Wenze Li1, Alexis J. Yagielski1, Hanzhi T. Zhao 1, Citlali Perez-Campos 1, Joyce M. Liu1, Elizabeth Philipone2, Angela J. Yoon 2, Kenneth P. Olive3, Shana M. Coley 4 and Elizabeth M. C. Hillman 1  1Laboratory for Functional Optical Imaging, Department of Biomedical Engineering and Radiology and the Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University 2Department of Oral and Maxillofacial Pathology, Columbia University Irving Medical Center 3Division of Digestive and Liver Disease, Herbert Irving Comprehensive Cancer Center, Columbia University Irving Medical Center 4Department of Pathology and Cell Biology, Vagelos College of Physicians and Surgeons, Columbia University Medical Center

Funding for this work was provided by the Columbia-Coulter Translational Research Partnership and the Coulter Foundation Early Career programme to E.M.C.H; the National Institutes of Health BRAIN initiative grants U01NS09429, UF1NS108213 to E.M.C.H and U19NS104649 to Costa; NCI grant U01CA236554 to E.M.C.H. and Brenner; the National Science Foundation NSF-GRFP DGE - 1644869 to K.B.P., IGERT 0801530 to V.V. and CAREER CBET-0954796 to E.M.C.H.; the Simons Foundation Collaboration on the Global Brain 542951 to E.M.C.H.; the Department of Defense MURI W911NF-12-1-0594 to E.M.C.H.; and the Kavli Institute for Brain Science to E.M.C.H.

COI: Intellectual property related to SCAPE microscopy is held by Columbia University and is licensed to Leica Microsystems for certain applications. The authors of this study could benefit financially from commercial development of this technology.