LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos

Yujun Shi^*    Jun Hao Liew^{*^}    Hanshu Yan    Vincent Y. F. Tan    Jiashi Feng
National University of Singapore   |   ByteDance

Disclaimer

This is a research project, NOT a commercial product. Users are granted the freedom to create images using this tool, but they are expected to comply with local laws and utilize it in a responsible manner. The developers do NOT assume any responsibility for potential misuse by users.

Update

• [2024.09.17] Release inference code and model.

Installation

See Installation for installation.

Gradio demo

Run gradio locally by

python3 drag_ui.py \
    --base_sd_path="checkpoints/dreamshaper-8-inpainting/" \
    --vae_path="checkpoints/sd-vae-ft-mse/" \
    --ip_adapter_path="checkpoints/IP-Adapter/models/" \
    --lightning_drag_model_path="checkpoints/lightning-drag-sd15" \
    --lcm_lora_path="checkpoints/lcm-lora-sdv1-5"

Please refer to the GIF above for step-by-step demo on how to use this UI.

IMPORTANT NOTES

• Since runwayml/stable-diffusion-inpainting is no longer available, we replace the inpainting checkpoint with Lykon/dreamshaper-8-inpainting without retraining or finetuning. Although it works, the results may not look the same as the one in the paper.
• To reproduce the results from the paper, you may need to replace base_sd_path with your own stable-diffusion-inpainting checkpoint.

Qualitative Results Gallery

Contact

For any questions on this project, please contact Yujun (shi.yujun@u.nus.edu) and Jun Hao (junhao.liew@bytedance.com)

BibTeX

If you find our repo helpful, please consider leaving a star or cite our paper :)

@article{shi2024lightningdrag,
         title={LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos},
         author={Shi, Yujun and Liew, Jun Hao, and Yan, Hanshu and Tan, Vincent YF and Feng, Jiashi},
         journal={arXiv preprint arXiv:2405.13722},
         year={2024}
}

Acknowledgement

Source image samples are collected from unsplash, pexels, pixabay. Also, a huge shout-out to all the amazing open source diffusion models, libraries, and technical reports.