Selected Publications

• OpticalNet: An Optical Imaging Dataset and Benchmark Beyond the Diffraction Limit
  Benquan Wang*, Ruyi An*, Jin-Kyu So, Sergei Kurdiumov, Eng Aik Chan, Giorgio Adamo, Yuhan Peng, Yewen Li, and Bo An.
  CVPR, 2025
  Abstract PDF Project Page BibTeX
     Observing objects of small size has always been a charming pursuit of human beings. However, due to the physical phenomenon of diffraction, the optical resolution is restricted to approximately half the wavelength of light, which impedes the observation of subwavelength objects, typically smaller than 200 nm. This constrains its application in numerous scientific and industrial fields that aim to observe objects beyond the diffraction limit, such as native state coronavirus inspection. Fortunately, deep learning methods have shown remarkable potential in uncovering underlying patterns within data, promising to overcome the diffraction limit by revealing the mapping pattern between diffraction images and their corresponding ground truth object localization images. However, the absence of suitable datasets has hindered progress in this field - collecting high-quality optical data of subwavelength objects is very challenging as these objects are inherently invisible under conventional microscopy, making it impossible to perform standard visual calibration and drift correction. Therefore, in collaboration with top optical scientists, we provide the first general optical imaging dataset based on the "LEGO" concept for addressing the diffraction limit. Drawing an analogy to the modular construction of the LEGO blocks, we construct a comprehensive optical imaging dataset comprising subwavelength fundamental elements, i.e., small square units that can be assembled into larger and more complex objects of any shape. We then frame the task as an image-to-image translation task and evaluate various vision backbone methods. Experimental results validate our "LEGO" concept, demonstrating that models trained on basic square units can effectively generalize to realistic, more complex unseen objects. Most importantly, by highlighting this underexplored AI-for-science area and its potential, we aspire to advance optical science by fostering collaboration with the vision and machine learning communities.
  @inproceedings{wang2025optical,
      title={Optical{Net}: An Optical Imaging Dataset and Benchmark Beyond the Diffraction Limit},
      author={Wang, Benquan and An, Ruyi, and So, Jin-Kyu and Kurdiumov, Sergei and Chan, Eng Aik and Adamo, Giorgio and Peng, Yuhan and Li, Yewen and An, Bo},
      booktitle={CVPR},
      year={2025}
  }
  
  
  
• Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection
  Yewen Li, Chaojie Wang, Xiaobo Xia, Xu He, Ruyi An, Dong Li, Tongliang Liu, Bo An and Xinrun Wang.
  arXiv Preprint, 2024
  Abstract PDF Code BibTeX
     Unsupervised out-of-distribution (U-OOD) detection is to identify OOD data samples with a detector trained solely on unlabeled in-distribution (ID) data. The likelihood function estimated by a deep generative model (DGM) could be a natural detector, but its performance is limited in some popular "hard" benchmarks, such as FashionMNIST (ID) vs. MNIST (OOD). Recent studies have developed various detectors based on DGMs to move beyond likelihood. However, despite their success on "hard" benchmarks, most of them struggle to consistently surpass or match the performance of likelihood on some "non-hard" cases, such as SVHN (ID) vs. CIFAR10 (OOD) where likelihood could be a nearly perfect detector. Therefore, we appeal for more attention to incremental effectiveness on likelihood, i.e., whether a method could always surpass or at least match the performance of likelihood in U-OOD detection. We first investigate the likelihood of variational DGMs and find its detection performance could be improved in two directions: i) alleviating latent distribution mismatch, and ii) calibrating the dataset entropy-mutual integration. Then, we apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. The final method, named Resultant, combines these two directions for better incremental effectiveness compared to either technique alone. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector while maintaining incremental effectiveness on likelihood in a wide range of tasks.
  @article{li2024resultant,
      title={Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection},
      author={Li, Yewen and Wang, Chaojie and Xia, Xiaobo and He, Xu and An, Ruyi and Li, Dong and Liu, Tongliang and An, Bo and Wang, Xinrun},
      journal={arXiv preprint arXiv:2409.03801},
      year={2024}
  }
  
  
  
• Cradle: Empowering foundation agents towards general computer control
  Weihao Tan*, Wentao Zhang*, Xinrun Xu*, ..., Ruyi An, ..., Bo An, Shuicheng Yan and Zongqing Lu. (27 authors)
  arXiv Preprint, 2024
  Abstract PDF Code BibTeX
     Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. With a unified interface to interact with any software, CRADLE greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents’ various abilities and facilitate further data collection, thus paving the way for generalist agents.
  @article{tan2024cradle,
      title={Cradle: Empowering Foundation Agents towards General Computer Control},
      author={Weihao Tan and Wentao Zhang and Xinrun Xu and Haochong Xia and Ziluo Ding and Boyu Li and Bohan Zhou and Junpeng Yue and Jiechuan Jiang and Yewen Li and Ruyi An and Molei Qin and Chuqiao Zong and Longtao Zheng and Yujie Wu and Xiaoqiang Chai and Yifei Bi and Tianbao Xie and Pengjie Gu and Xiyun Li and Ceyao Zhang and Long Tian and Chaojie Wang and Xinrun Wang and Börje F. Karlsson and Bo An and Shuicheng Yan and Zongqing Lu},
      journal={arXiv preprint arXiv:2403.03186},
      year={2024}
  }
  
  
  
• Improving Unsupervised Hierarchical Representation with Reinforcement Learning
  Ruyi An, Yewen Li, Xu He, Pengjie Gu, Mengchen Zhao, Dong Li, Jianye Hao, Chaojie Wang, Bo An and Mingyuan Zhou.
  CVPR 2024
  Abstract PDF Code BibTeX
     Learning representations to capture the very fundamental understanding of the world is a key challenge in machine learning. The hierarchical structure of explanatory factors hidden in data is such a general representation and could be potentially achieved with a hierarchical VAE. However training a hierarchical VAE always suffers from the "posterior collapse" where the data information is hard to propagate to the higher-level latent variables hence resulting in a bad hierarchical representation. To address this issue we first analyze the shortcomings of existing methods for mitigating the "posterior collapse" from an information theory perspective then highlight the necessity of regularization for explicitly propagating data information to higher-level latent variables while maintaining the dependency between different levels. This naturally leads to formulating the inference of the hierarchical latent representation as a sequential decision process which could benefit from applying reinforcement learning (RL). Aligning RL's objective with the regularization we first introduce a "skip-generative path" to acquire a reward for evaluating the information content of an inferred latent representation and then the developed Q-value function based on it could have a consistent optimization direction of the regularization. Finally policy gradient one of the typical RL methods is employed to train a hierarchical VAE without introducing a gradient estimator. Experimental results firmly support our analysis and demonstrate that our proposed method effectively mitigates the "posterior collapse" issue learns an informative hierarchy acquires explainable latent representations, and significantly outperforms other hierarchical VAE-based methods in downstream tasks.
  @inproceedings{hier_rep_rl,
      title={Improving Unsupervised Hierarchical Representation with Reinforcement Learning},
      author={An, Ruyi and Li, Yewen and He, Xu and Gu, Pengjie and Zhao, Mengchen and Li, Dong and Hao, Jianye and An, Bo and Wang, Chaojie and Zhou, Mingyuan},
      booktitle={{CVPR}},
      year={2024}
  }
  
  
  
• Retrieving positions of closely packed sub-wavelength nanoparticles from their diffraction patterns
  Benquan Wang, Ruyi An, Eng Aik Chan, Giorgio Adamo, Jin-Kyu So, Yewen Li, Zexiang Shen, Bo An and Nikolay I. Zheludev.
  Applied Physics Letter, 2023 (APL).
  Abstract PDF BibTeX
  
     Distinguishing two objects or point sources located closer than the Rayleigh distance is impossible in conventional microscopy. Understandably, the task becomes increasingly harder with a growing number of particles placed in close proximity. It has been recently demonstrated that subwavelength nanoparticles in closely packed clusters can be counted by AI-enabled analysis of the diffraction patterns of coherent light scattered by the cluster. Here we show that deep learning analysis can determine the actual position of the nanoparticle in the cluster of subwavelength particles from a sing-shot diffraction pattern even if they are separated by distances below the Rayleigh resolution limit of a conventional microscope.
  
  
  
  @article{wang2023retriving,
      title={Retrieving positions of closely packed sub-wavelength nanoparticles from their diffraction patterns},
      author={Wang, Benquan and An, Ruyi and Chan, Eng Aik and Adamo, Giorgio and So, Jin-Kyu and Li, Yewen and Shen, Zexiang and An, Bo and Zheludev, Nikolay I.},
      journal={Applied Physics Letters},
      volume={124},
      number={15},
      year={2024},
      publisher={AIP Publishing}
  }
  
  
  
  
  
• Continuous Emotion Recognition Using Visual-Audio-Linguistic Information
  Su Zhang, Ruyi An, Yi Ding and Cuntai Guan.
  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  PDF Code BibTeX
  
  @inproceedings{zhang2022continuous,
      title={Continuous emotion recognition using visual-audio-linguistic information: A technical report for abaw3},
      author={Zhang, Su and An, Ruyi and Ding, Yi and Guan, Cuntai},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
      pages={2376--2381},
      year={2022}
  }