fyi you can speed up rendering by ~50%. Replace https://github.com/rgl-epfl/large-steps-pytorch/blob/e03b40e237276b0efe32d022f5886b81db45bc3c/scripts/render.py#L210-L213 by

vert_light = self.sh.eval(n).contiguous()
light = dr.interpolate(vert_light[None, ...], rast, f)[0]

I've just cloned the project and I'm trying to run the Tutorial.

The whole directory scenes/suzanne and the suzanne.xml is missing

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-7-8c8db06f0156> in <module>
      1 # Load the scene
      2 filepath = os.path.join(os.getcwd(), "scenes", "suzanne", "suzanne.xml")
----> 3 scene_params = load_scene(filepath)
      4 
      5 # Load reference shape

~/Desktop/large-steps-pytorch/scripts/load_xml.py in load_scene(filepath)
     58     assert ext == ".xml", f"Unexpected file type: '{ext}'"
     59 
---> 60     tree = ET.parse(filepath)
     61     root = tree.getroot()
     62 

~/miniconda3/envs/pytorch3d_06/lib/python3.8/xml/etree/ElementTree.py in parse(source, parser)
   1200     """
   1201     tree = ElementTree()
-> 1202     tree.parse(source, parser)
   1203     return tree
   1204 

~/miniconda3/envs/pytorch3d_06/lib/python3.8/xml/etree/ElementTree.py in parse(self, source, parser)
    582         close_source = False
    583         if not hasattr(source, "read"):
--> 584             source = open(source, "rb")
    585             close_source = True
    586         try:

FileNotFoundError: [Errno 2] No such file or directory: '/home/bobi/Desktop/large-steps-pytorch/scenes/suzanne/suzanne.xml'

I try to build this project on my windows for a week, and unfortunately failed, can you give me the specific process of build the project on windows? :) The failure I met is related to libs in the ext/ (mainly numpyeigen).

Hi, I got problems after updating to the latest 0.2.0 version. When my program invoked the from_differential function, it got stuck for a little while, and then exited directly. Nothing (warnings/errors/...) was shown on my prompt, and so I could not figure out what happened. However, the initial 0.1.1 version worked well. Tested on: Windows 10, AMD Ryzen 9 5900HX with Radeon Graphics @ 3.30GHz 16GB, GeForce RTX 3070 Laptop GPU 8GB.

GPU memory is not properly freed when switching to other meshes, eventually leading to CUSPARSE_STATUS_ALLOC_FAILED:

Traceback (most recent call last):
  File "scripts/show_largesteps_memory_leak.py", line 16, in <module>
    v = from_differential(M, u, 'Cholesky')
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/parameterize.py", line 51, in from_differential
    solver = CholeskySolver(L)
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/solvers.py", line 130, in __init__
    self.solver_1 = prepare(self.L, False, False, True)
  File "/home/xuzhen/miniconda3/envs/flame/lib/python3.8/site-packages/largesteps/solvers.py", line 68, in prepare
    _cusparse.scsrsm2_analysis(
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2103, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2115, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1511, in cupy_backends.cuda.libs.cusparse.check_status
cupy_backends.cuda.libs.cusparse.CuSparseError: CUSPARSE_STATUS_ALLOC_FAILED

To reproduce, run this code example with this example mesh (extract armadillo.npz and place it where you run the code below):

import torch
import numpy as np
from tqdm import tqdm

from largesteps.parameterize import from_differential, to_differential
from largesteps.geometry import compute_matrix
from largesteps.optimize import AdamUniform

armadillo = np.load('armadillo.npz')
verts = torch.tensor(armadillo['v'], device='cuda')
faces = torch.tensor(armadillo['f'], device='cuda')

for i in tqdm(range(3000)):
    # assume there's different meshes w/ different topology
    M = compute_matrix(verts, faces, 10)
    u = to_differential(M, verts)
    u.requires_grad_()
    optim = AdamUniform([u], 3e-2)
    for j in range(5):
        v: torch.Tensor = from_differential(M, u, 'Cholesky')
        loss: torch.Tensor = (v.norm(dim=-1) - 1).mean()
        optim.zero_grad()
        loss.backward()
        optim.step()

While running the code above, you should see the GPU memory continuously increase but the expected behavior is that it stays constant.

For example, the result of nvidia-smi dmon -s m while running the code should be something like:

Hi,

Thank you for your excellent work, but I have a question. How should I handle my data so that it can be accepted by this framework? I only have meshes. I do not have the .blender file and the .xml file. And I have texture files of different formart.

Thank you.

I compiled and started the project, watches videos and papers but still can't understand the purpose of project. Is it reconstruction from images or this solutions solving one of the problems of area with reconstruction from images? In sources I can see source and destination model no images. Is this method showing how to get the same model like in target with simple in source but not from images? Is I understand correctly: The project giving target model -> render it from different positions and using this images for reconstruct the scene back? If the method using light of areas for reconstruct normals and mesh how far is it from using with photos from real life?

First of all I'd like to thank you for making this phenomenal experience and make it available for testing. Running the nvdiffrast can be really straight forward and easy for rendering but I'm having some understanding problem with rendering the code inside the blender so please bear with me :)

How do I run the rendering inside the blender? Is it by script editor? [I tried it but it gives me errors] Is it by running it as command ?

I just need to understand the methodology of rendering that in blender since it's a utility and not included in the Tutorial.

Thank you so much.

Hi, Thanks for sharing your work! I tried to use the Tutorial notebook on the Dragon mesh but get really poor results. Can you share the parameters you used to make the model converge? thanks a lot

Hi Baptiste, thanks for publishing the code:-) I found two requirements missing in the installation instructions:

• apt install libeigen3-dev (required for building botsch-kobbelt)
• conda install cudatoolkit-dev -c conda-forge (required by nvdiffrast)

Not sure if this should be solved here, in cholespy, or nanobind. The from_differential function throws an error if the second argument is a torch.nn.Parameter rather than a tensor. Parameter is directly derived from Tensor, so there's no reason the cast should fail.

TypeError: solve(): incompatible function arguments. The following argument types are supported:
    1. solve(self, b: tensor[dtype=float32, order='C'], x: tensor[dtype=float32, order='C']) -> None

Invoked with types: CholeskySolverF, Parameter, Tensor

It's quite hard to workaround this "from the outside". E.g. doing from_differential(M, x.data) doesn't work because the gradient will be written to x.data.grad whereas the optimizer expects x.grad.

Need to check whether there are gradients or not when updating parameters. Ref: https://github.com/pytorch/pytorch/blob/d05f07494a9a32c63f9218c0e703764a02033bb9/torch/optim/adam.py#L134