Importing texture + image visual#34
Conversation
|
|
Do you have some specs or notes on Texture and Image ? |
|
@rougier im not sure i understand what you mean... Do you want a longer description for the PR ? something else ? |
|
Texture can be actually 1D, 2D, 3D and the docstring on your texture objects implies it is 2D. In the meantime I can merge if we want to test your implementation. |
|
Indeed, in Datoviz, one must explicitly specify 1D, 2D, or 3D when creating a texture. |
|
@rougier Here the image is always facing the camera. what we call a @rossant i can create more specific class if needed. Do you guys want the same visual 'image facing the camera' and 'oriented polygon with a texture' ? this seems a quite a different thing. what about we call this one PS: how to display a texture in matplotlib ? i have seen this https://github.com/rougier/tiny-renderer/blob/master/head.py .. it run directly on the |
|
|
I think Planar image + orthognal axis might be enough for image. This is used for example when projecting iamge on 3D cube. Such projection is not straighforward with matplotlib though. I will post my experimental code below. |
You're right the two are quitte different. In Datoviz, |
So this is not a sprite as this image may not be facing camera, right? |
# Package: Graphic Server Protocol / Matplotlib
# Authors: Nicolas P .Rougier <nicolas.rougier@inria.fr>
# License: BSD 3 clause
import glm
import camera
import numpy as np
import imageio.v3 as iio
import matplotlib.pyplot as plt
from matplotlib.path import Path
import matplotlib.transforms as mtransforms
from matplotlib.patches import PathPatch
from matplotlib.collections import PolyCollection
def warp(T1, T2):
"""
Return an affine transform that warp triangle T1 into triangle
T2.
Parameters
----------
T1 : (3,2) np.ndarray
Positions of the first triangle vertices
T2 : (3,2) np.ndarray
Positions of the first triangle vertices
Raises
------
`LinAlgError` if T1 or T2 are degenerated triangles
"""
T1 = np.c_[np.array(T1), np.ones(3)]
T2 = np.c_[np.array(T2), np.ones(3)]
M = np.linalg.inv(T1) @ T2
return mtransforms.Affine2D(M.T)
def textured_triangle(ax, T, UV, texture, intensity,
interpolation="none", image=None, zorder=0):
"""
Parameters
----------
T : (3,2) np.ndarray
Positions of the triangle vertices
UV : (3,2) np.ndarray
UV coordinates of the triangle vertices
texture:
Image to use for texture
"""
w,h = texture.shape[:2]
Z = UV*(w,h)
xmin, xmax = int(np.floor(Z[:,0].min())), int(np.ceil(Z[:,0].max()))
ymin, ymax = int(np.floor(Z[:,1].min())), int(np.ceil(Z[:,1].max()))
texture = (texture[ymin:ymax, xmin:xmax,:] * intensity).astype(np.uint8)
extent = xmin/w, xmax/w, ymin/h, ymax/h
transform = warp (UV,T) + ax.transData
path = Path([UV[0], UV[1], UV[2], UV[0]], closed=True)
if image is not None:
image.set_data(texture)
image.set_extent(extent)
image.set_transform(transform)
image.set_clip_path((path,transform))
image.patch.set_path(path)
image.patch.set_transform(transform)
else:
image = ax.imshow(texture, interpolation=interpolation, origin='lower',
zorder=zorder,
extent=extent, transform=transform, clip_path=(path,transform))
patch = PathPatch(path, facecolor="none", edgecolor=(0.0,0.0,0.0,0.5),
linewidth=0.25, transform=transform, zorder=zorder+1)
image.patch = patch
ax.add_patch(patch)
return image
def obj_read(filename):
"""
Read a wavefront filename and returns vertices, texcoords and
respective indices for faces and texcoords
"""
V, T, N, Vi, Ti, Ni = [], [], [], [], [], []
with open(filename) as f:
for line in f.readlines():
if line.startswith('#'):
continue
values = line.split()
if not values:
continue
if values[0] == 'v':
V.append([float(x) for x in values[1:4]])
elif values[0] == 'vt':
T.append([float(x) for x in values[1:3]])
elif values[0] == 'vn':
N.append([float(x) for x in values[1:4]])
elif values[0] == 'f' :
Vi.append([int(indices.split('/')[0]) for indices in values[1:]])
Ti.append([int(indices.split('/')[1]) for indices in values[1:]])
Ni.append([int(indices.split('/')[2]) for indices in values[1:]])
return np.array(V), np.array(T), np.array(Vi)-1, np.array(Ti)-1
from gsp import core, visual, transform, glm
canvas = core.Canvas()
viewport = core.Viewport(canvas)
camera = camera.Camera("perspective", theta=10, phi=-5)
positions, texcoords, face_indices, texcoords_indices = obj_read("data/head.obj")
texture = iio.imread("data/uv-grid.png")[::-1,::1,:3]
images = []
def update(viewport=None, model=None, view=None, proj=None):
transform = proj @ view @ model
V = positions[face_indices]
UV = texcoords[texcoords_indices]
# Computer lighting on non-projected faces
N = np.cross(V[:,2]-V[:,0], V[:,1]-V[:,0])
N = N / np.linalg.norm(N,axis=1).reshape(len(N),1)
L = np.dot(N, (0,0,-1))
V = glm.to_vec3(glm.to_vec4(positions) @ transform.T)
V = V[face_indices]
I = np.argsort(-V[:,:,2].mean(axis=1))
V = V[I][...,:2]
UV = UV[I][...,:2]
L = abs(L[I])
ax = viewport._axes
zorder = 10
if not len(images):
for v, uv, l in zip(V, UV, L):
if l > 0:
try:
image = textured_triangle(ax, v, uv, texture, (l+1)/2, "none", None, zorder)
images.append(image)
except np.linalg.LinAlgError:
pass
zorder += 2
else:
for v, uv, l, image in zip(V, UV, L, images):
if l > 0:
try:
image = textured_triangle(ax, v, uv, texture, (l+1)/2, "none", image, zorder)
except np.linalg.LinAlgError:
pass
zorder += 2
camera.connect(viewport, "motion", update)
update(viewport, camera.model, camera.view, camera.proj)
# plt.savefig("head-camera.pdf")
plt.show() |
|
@cyrille Yes, the image will face camera if no axis is given and is oriented in space when an axis is given. This is what I ddi for markers. |
Okay. Perhaps this is something Datoviz could support in the Image visual, I imagine it could be relatively easy (just rotating the quad vertices in 3D). |
|
@rougier can you commit a working version of that somewhere ? |
|
Not easily, it is wit a previous experimental code. The important function is the textured triangle function that shoud lwork as expected. And yes, it is slow. |
|
@rougier can you run this code ? |
|
@rougier thanks |
|
Note that we cannor avoid the thin line around triangles because of antialiasing that cannot be removed when a clip mask is used. |
Proposal on naming
|
|
In Datoviz, I will probably use the same Image visual for both, as it is just a matter of adding a couple of arguments to the existing Image visual (axis and possible rotation angle around it). In the Datoviz GSP renderer, I should therefore have the information whether I am receiving a camera-facing image, or 3D image, or a generic mesh. With your proposal, would I be able to know which of these three cases I'm in? |
|
I would prefer a (planar) Image visual with an axis option for orientation.This already exists for Markers. Now, concerning the implementation, it is actually a Mesh but Images are so common in sciviz it might be worth to have a dedicated visual. |
|
The image shader is certainly much simpler than the mesh shader, which supports lighting etc. So yes, it would make sense to have a separate visual for the mesh. And an Image visual with an optional axis vector would work, I think. @rougier do we also need an angle argument in addition to the axis? |
for the renderer point of view, it will be either camera-facing image or a mesh. there is no such thing as a 3d image (quad plane orientated in 3d).
|
so you suggest to have twice the code of mesh ? like once in image, and once in mesh on that i will quote rfc 1925 😃 a very good read
Another quote from unix philosophy : "do one thing and do it well" |
|
Still, Images & Meshes are not exactly the same since. |
jeromeetienne
left a comment
jeromeetienne
left a comment
There was a problem hiding this comment.
@rougier i dont get it, willing to show how to display an image in 3d on matplotlib, you provided a mesh display.
isn't that a hint that those 2 code are similar ?
|
The code may be the same in matplotlib, but not in Datoviz, so separating the two in the protocol may make sense? Is it possible to use the same code path in the matplotlib GSP renderer, for the two different visual abstractions? I agree code should not be duplicated! But I think it's possible to avoid duplication while keeping a "virtual" distinction at the protocol spec level. |
|
The mesh / image code will be approximately the same for matplotlib but this is not a good reason to not have a Image and a Mesh for GSP. At the extreme point, everything can be rendered with only triangles and still, we offer higher level API. Image is such an example because it is pervasive in sciviz. The way it is implemented is another story. In matplotlib, you could use the Mesh visual under the hood. |
jeromeetienne
left a comment
jeromeetienne
left a comment
There was a problem hiding this comment.
yes we can aglutenate multiple separate problems into a single complex interdependent solution. it is possible to do so.
It will result is hard to maintain code, and will be error prone.
i will do it, please provide a clear definition of the API you want.
|
Well, on the contrary, I think it is better to split the complex mesh visual into multiple separate problems, the generic mesh, and the specific orientable image in 3D. If you prefer, we could also split the Image visual in two:
That would make three different visuals at the API level:
@rougier @jeromeetienne what do you think? |
|
Maybe we should start by defining the API, independently of implementation. |
jeromeetienne
left a comment
jeromeetienne
left a comment
There was a problem hiding this comment.
Let's have some user stories too, thus we can justify the tech choices
3 participants
This PR adds the texture + images
Notes
__init__.pyto include the new files texture+imageimage.render()function