Saaaaln opened a new issue, #14619: URL: https://github.com/apache/tvm/issues/14619
Recently, I try to compile my model using [torchdynamo with tvm backend](https://pytorch.org/docs/stable/dynamo/get-started.html). But I get the following errors: ``` /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") /mnt/petrelfs/code/tvm/python/tvm/target/target.py:397: UserWarning: Try specifying cuda arch by adding 'arch=sm_xx' to your target. warnings.warn("Try specifying cuda arch by adding 'arch=sm_xx' to your target.") Traceback (most recent call last): File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/fx/graph_module.py", line 271, in __call__ return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1488, in _slow_forward result = self.forward(*input, **kwargs) File "<eval_with_key>.13", line 7, in forward _stack0[_stack1] = ior; setitem = _stack0; _stack0 = _stack1 = ior = None RuntimeError: shape mismatch: value tensor of shape [802816] cannot be broadcast to indexing result of shape [0] Call using an FX-traced Module, line 7 of the traced Module's generated forward function: _stack2 |= invert; ior = _stack2; _stack2 = invert = None _stack0[_stack1] = ior; setitem = _stack0; _stack0 = _stack1 = ior = None ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE invert_1 = ~ray_invalid; ray_invalid = None zeros = torch.zeros((16384, 128), device = device(type='cuda', index=0)) 0%| | 0/5 [00:39<?, ?it/s] Traceback (most recent call last): File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler compiled_fn = compiler_fn(gm, self.fake_example_inputs()) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper compiled_gm = compiler_fn(gm, example_inputs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/backends/common.py", line 107, in wrapper return fn(model, inputs, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/backends/tvm.py", line 22, in tvm jit_mod = torch.jit.trace(gm, example_inputs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn return fn(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/jit/_trace.py", line 794, in trace return trace_module( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/jit/_trace.py", line 1056, in trace_module module._c._create_method_from_trace( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/fx/graph_module.py", line 662, in call_wrapped return self._wrapped_call(self, *args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/fx/graph_module.py", line 279, in __call__ raise e.with_traceback(None) RuntimeError: shape mismatch: value tensor of shape [802816] cannot be broadcast to indexing result of shape [0] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/renderer.py", line 42, in <module> start_render(args) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/renderer.py", line 14, in start_render render_test(render_args) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/commons/render_base.py", line 361, in render_test PSNRs_test = evaluation( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/commons/render_base.py", line 91, in evaluation all_ret, _ = renderer( File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/commons/render_base.py", line 30, in renderer_fn ret, extra_loss = tensorf( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn return fn(*args, **kwargs) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/modules/models/tensorBase.py", line 492, in forward xyz_sampled, z_vals, ray_valid = self.sample_ray_within_hull( File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/modules/models/tensorBase.py", line 505, in <graph break in forward> alphas = self.alphaMask.sample_alpha(xyz_sampled[ray_valid], profiler=profiler) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/modules/models/tensorBase.py", line 505, in <graph break in forward> alphas = self.alphaMask.sample_alpha(xyz_sampled[ray_valid], profiler=profiler) File "/mnt/petrelfs/new_code/landmarks/gridnerf/render/python/modules/models/tensorBase.py", line 508, in <graph break in forward> ray_invalid[ray_valid] |= ~alpha_mask File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors return callback(frame, cache_size, hooks) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame result = inner_convert(frame, cache_size, hooks) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn return fn(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert return _compile( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper r = func(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile out_code = transform_code_object(code, transform) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object transformations(instructions, code_options) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform tracer.run() File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run super().run() File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run and self.step() File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step getattr(self, inst.opname)(inst) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 269, in inner self.output.compile_subgraph( File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 517, in compile_subgraph self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph compiled_fn = self.call_user_compiler(gm) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper r = func(*args, **kwargs) File "/mnt/petrelfs/anaconda3/envs/my_nerff/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler raise BackendCompilerFailed(self.compiler_fn, e) from e torch._dynamo.exc.BackendCompilerFailed: tvm raised RuntimeError: shape mismatch: value tensor of shape [802816] cannot be broadcast to indexing result of shape [0] You can suppress this exception and fall back to eager by setting: torch._dynamo.config.suppress_errors = True ``` ### Expected behavior There is my code to be: ``` // ray_invalid.shape: torch.Size([16384, 128]) ray_invalid = ~ray_valid // ray_invalid[ray_valid].shape: torch.Size([802816]) // ~alpha_mask.shape: torch.Size([802816]) ray_invalid[ray_valid] |= ~alpha_mask ray_valid = ~ray_invalid ``` The shape of `ray_invalid[ray_valid] ` and `alpha_mask` both are 802816. It should not be broadcast to indexing result of shape [0]. ### Environment GCC 11.2.0 CentOS 7.6.1810 TVM version: build from source code. commit: `fc2a9e50afd03adba9839b94ef6d10145be5fc37` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
