Author Topic: Hardware skinning needed  (Read 29745 times)

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #15 on: May 27, 2014, 04:51:38 pm »
I can't use SkinClipSequence animate() method as it's a private method. Could you compile a new Bones.jar with the method declared as PUBLIC? This's the best I can think of. Would love to hear if there is better suggestion ;)

Offline raft

  • Moderator
  • quad
  • *****
  • Posts: 1993
    • View Profile
    • http://www.aptalkarga.com
Re: Hardware skinning needed
« Reply #16 on: May 27, 2014, 05:05:18 pm »
you have the source code, just change it and recompile. the zip even contains the ant script to build the whole project ;)

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #17 on: May 27, 2014, 05:16:23 pm »
Ah~ I missed out the source code in the zip file. I always thought it's close sourced. Ok, i can start working on it now. Thanks alot.

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #18 on: June 07, 2014, 10:13:31 am »
Hi Raft,

I've spending few weeks to implemet hardware skinning, but I encountered few problems and I hope you could help me out on this

I've made these code, but the result seems incorrect:

vertex shader: skinPalette is hardcoded to match the bones file I have (im not sure if the number of matrix palettes is fixed).
Code: [Select]
uniform mat4 modelViewMatrix;
uniform mat4 modelViewProjectionMatrix;
uniform mat4 textureMatrix;

uniform vec4 additionalColor;
uniform vec4 ambientColor;

uniform float alpha;
uniform float shininess;
uniform bool useColors;

uniform float fogStart;
uniform float fogEnd;
uniform vec3 fogColor;

uniform int lightCount;

uniform mat4 skinPalette[7];

uniform vec3 lightPositions[8];
uniform vec3 diffuseColors[8];
uniform vec3 specularColors[8];
uniform float attenuation[8];

attribute vec4 position;
attribute vec3 normal;
attribute vec4 color;
attribute vec2 texture0;
attribute vec2 texture1;
attribute vec2 texture2;
attribute vec2 texture3;

attribute vec4 skinWeights;
attribute vec4 skinJointIndices;

varying vec2 texCoord[4];
varying vec4 vertexColor;
varying vec3 fogVertexColor;
varying float fogWeight;

const vec4 WHITE = vec4(1,1,1,1);

void main() {

texCoord[0] = texture0;
texCoord[1] = (textureMatrix * vec4(texture1, 0, 1)).xy;
texCoord[2] = texture2;
texCoord[3] = texture3;

vec4 myPosition = vec4(position);
vec3 myNormal = vec3(normal);

vec4 vertexTemp;
vec3 normalTemp;
mat4 mat;


for (int i = 0; i < 4; i++) {
mat = skinPalette[int(skinJointIndices[i])];
vertexTemp = mat * position;
vertexTemp *= skinWeights[i];
myPosition += vertexTemp;
normalTemp = normal * mat3(mat);
normalTemp *= skinWeights[i];
myNormal += normalTemp;
}

vec4 vertexPos = modelViewMatrix * myPosition;
vertexColor = ambientColor + additionalColor;

if (lightCount>0) {
// This is correct only if the modelview matrix is orthogonal. In jPCT-AE, it always is...unless you fiddle around with it.
vec3 normalEye   = normalize(modelViewMatrix * vec4(myNormal, 0.0)).xyz;

float angle = dot(normalEye, normalize(lightPositions[0] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[0] * angle + specularColors[0] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[0] - vertexPos.xyz)*attenuation[0])), 1);
}

// Freaky Adreno shader compiler can't handle loops without locking or creating garbage results....this is why the
// loop has been unrolled here. It's faster this way on PowerVR SGX540 too, even if PVRUniSCoEditor says otherwise...

if (lightCount>1) {
angle = dot(normalEye, normalize(lightPositions[1] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[1] * angle + specularColors[1] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[1] - vertexPos.xyz)*attenuation[1])), 1);
}

}
}


if (fogStart != -1.0) {
fogWeight = clamp((-vertexPos.z - fogStart) / (fogEnd - fogStart), 0.0, 1.0);
fogVertexColor = fogColor * fogWeight;
} else {
fogWeight = -1.0;
}

vertexColor=vec4(min(WHITE, vertexColor).xyz, alpha);

if (useColors) {
vertexColor *= color;
}

gl_Position = modelViewProjectionMatrix * myPosition;
}

This's the code to add vertex attributes to shader. linearizeArray is converting float[][] to float[] and linearizeToFloatArray is short[][] to float[]
Code: [Select]
float[] f1 = SkinHelper.linearizeArray(skin.weights);
float[] f2 = SkinHelper.linearizeToFloatArray(skin.jointIndices);
skinWeights = new VertexAttributes(
"skinWeights", f1, VertexAttributes.TYPE_FOUR_FLOATS);
skinJointIndices = new VertexAttributes(
"skinJointIndices", f2, VertexAttributes.TYPE_FOUR_FLOATS);
Mesh mesh = getMesh();
mesh.addVertexAttributes(skinWeights);
mesh.addVertexAttributes(skinJointIndices);

This's where I set the uniform for matrix palette (in array form, Egon created a new GLSLShader to pass matrix array).
Code: [Select]
object3D.setRenderHook(new RenderHookAdapter(){

@Override
public void setCurrentShader(GLSLShader shader) {
super.setCurrentShader(shader);
shader.setUniform("skinPalette", object3D.getSkeletonPose().getPalette());
}
});

I tried to debug glsl and most debugger I tried were not working. Hope you could help me out on this. May be I missed something.

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #19 on: June 07, 2014, 11:42:53 am »
I got it worked!! I fixed the previous code with these:

Code: [Select]
uniform mat4 modelViewMatrix;
uniform mat4 modelViewProjectionMatrix;
uniform mat4 textureMatrix;

uniform vec4 additionalColor;
uniform vec4 ambientColor;

uniform float alpha;
uniform float shininess;
uniform bool useColors;

uniform float fogStart;
uniform float fogEnd;
uniform vec3 fogColor;

uniform int lightCount;

uniform mat4 skinPalette[7];

uniform vec3 lightPositions[8];
uniform vec3 diffuseColors[8];
uniform vec3 specularColors[8];
uniform float attenuation[8];

attribute vec4 position;
attribute vec3 normal;
attribute vec4 color;
attribute vec2 texture0;
attribute vec2 texture1;
attribute vec2 texture2;
attribute vec2 texture3;

attribute vec4 skinWeights;
attribute vec4 skinJointIndices;

varying vec2 texCoord[4];
varying vec4 vertexColor;
varying vec3 fogVertexColor;
varying float fogWeight;

const vec4 WHITE = vec4(1,1,1,1);

void main() {

texCoord[0] = texture0;
texCoord[1] = (textureMatrix * vec4(texture1, 0, 1)).xy;
texCoord[2] = texture2;
texCoord[3] = texture3;

vec4 myPosition = vec4(0,0,0,0);
vec3 myNormal = vec3(0,0,0);

vec4 vertexTemp;
vec3 normalTemp;
mat4 mat;

for (int i = 0; i < 4; i++) {
mat = skinPalette[int(skinJointIndices[i])];
vertexTemp = mat * position;
vertexTemp *= skinWeights[i];
myPosition += vertexTemp;
normalTemp = mat3(mat) * normal;
normalTemp *= skinWeights[i];
myNormal += normalTemp;
}

vec4 vertexPos = modelViewMatrix * myPosition;
vertexColor = ambientColor + additionalColor;

if (lightCount>0) {
// This is correct only if the modelview matrix is orthogonal. In jPCT-AE, it always is...unless you fiddle around with it.
vec3 normalEye   = normalize(modelViewMatrix * vec4(myNormal, 0.0)).xyz;

float angle = dot(normalEye, normalize(lightPositions[0] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[0] * angle + specularColors[0] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[0] - vertexPos.xyz)*attenuation[0])), 1);
}

// Freaky Adreno shader compiler can't handle loops without locking or creating garbage results....this is why the
// loop has been unrolled here. It's faster this way on PowerVR SGX540 too, even if PVRUniSCoEditor says otherwise...

if (lightCount>1) {
angle = dot(normalEye, normalize(lightPositions[1] - vertexPos.xyz));

if (angle > 0.0) {
vertexColor += vec4((diffuseColors[1] * angle + specularColors[1] * pow(angle, shininess))*(1.0/(1.0+length(lightPositions[1] - vertexPos.xyz)*attenuation[1])), 1);
}

}
}


if (fogStart != -1.0) {
fogWeight = clamp((-vertexPos.z - fogStart) / (fogEnd - fogStart), 0.0, 1.0);
fogVertexColor = fogColor * fogWeight;
} else {
fogWeight = -1.0;
}

vertexColor=vec4(min(WHITE, vertexColor).xyz, alpha);

if (useColors) {
vertexColor *= color;
}

gl_Position = modelViewProjectionMatrix * myPosition;
}

where myPosition and myNormal are initialized to zero.  But the normal seems to be abit off. I'm looking into it.

Offline raft

  • Moderator
  • quad
  • *****
  • Posts: 1993
    • View Profile
    • http://www.aptalkarga.com
Re: Hardware skinning needed
« Reply #20 on: June 07, 2014, 07:51:30 pm »
cool, i'm glad you solved it :)

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 12295
    • View Profile
    • http://www.jpct.net
Re: Hardware skinning needed
« Reply #21 on: June 07, 2014, 09:57:07 pm »
And how much faster is it now?

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #22 on: June 08, 2014, 07:53:11 am »
Quote
And how much faster is it now?
I tested it out by just doing a simple check on the time taken for the entire drawframe method. Test case is a model w/o texture with ~1968 vertices and 7 bones. Only 3 bones are moving. Test device is Samsung Galaxy S4 with octa core.

Here's the result:
Software: ~ 11-14 ms
Hardware skinning: ~ 1-4 ms

The test result seems to be quite promising. Anyway the rotation seems to be incorrect compared to the software version. I'll try check it out.
« Last Edit: June 08, 2014, 07:58:16 am by kkl »

Offline raft

  • Moderator
  • quad
  • *****
  • Posts: 1993
    • View Profile
    • http://www.aptalkarga.com
Re: Hardware skinning needed
« Reply #23 on: June 09, 2014, 09:13:40 am »
Quote from: kkl
The test result seems to be quite promising.
really they are :)

Offline kiffa

  • long
  • ***
  • Posts: 199
    • View Profile
Re: Hardware skinning needed
« Reply #24 on: June 23, 2014, 05:18:12 am »
Cool~ Would you like to share it when you done? This will give  others(include me  ;) ) a lot of help.

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #25 on: June 24, 2014, 03:40:59 pm »
Hi Kiffa,

It's almost done, but there's still some problem wif the initial rotation and I didn't have time to fix that yet. It would be great if you could fix it.

The attachment contains the required java files. You would need to download the Bones source code and replace the java files with the same names.

Example usage:
Code: [Select]
try {
group = BonesIO.loadGroup(context.getAssets().open(
"models/monkey.bones"));
} catch (Exception ex) {
ex.printStackTrace();
}

for (final Animated3D o : group) {

try {
final GLSLShader shader1 = new GLSLShader(
Loader.loadTextFile(context.getAssets().open("shaders/vertex_hardware_skinning.src")),
Loader.loadTextFile(context.getAssets().open("shaders/fragment.src")));
o.setShader(shader1);
o.setRenderHook(new RenderHookAdapter(){

@Override
public void beforeRendering(int arg0) {
shader1.setUniform("skinPalette", o.getSkeletonPose().getPalette());
}

});
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
o.setHardwareEnabled(true);
o.build();
o.discardMeshData();
}

Currently, I hard coded the vertex shader to support up to 10 bones only. If you wanna increase it, edit skinPalette[10] to bigger size array in vertex_hardware_skinning.src. The vertex shader also supports up to 2 light sources only, but you can go ahead and copy part of jpct original vertex shader and paste it to increase light sources.

[attachment deleted by admin]

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #26 on: April 13, 2015, 04:29:28 pm »
Hi raft and egon,

Sorry for bringing this up. Previously I have implemented hardware skinning and was having initial rotation issue, in which the object appears different rotation than the one original with software skinning. The incorrect rotation started to cause issue when JPCT detects clipping boundary and object is not in the screen and it's automatically not showing the object, but in actual it should show as it's just having different rotation after processing in shader, which is after clipping process.

I was just wondering if there's any addition rotations applied before vertex is passed to shader. I'm afraid I might have missed some rotation matrices, or it could be something else that cause the issue. Thanks.

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 12295
    • View Profile
    • http://www.jpct.net
Re: Hardware skinning needed
« Reply #27 on: April 13, 2015, 07:22:05 pm »
No, actually not. All vertex transformations happen inside the shader.

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #28 on: April 14, 2015, 02:45:46 pm »
Not sure where went wrong.. Currently i cant find the root cause, is it possbile if we can have an option to disable clipping object process for temporary solution?

Offline kkl

  • float
  • ****
  • Posts: 291
    • View Profile
Re: Hardware skinning needed
« Reply #29 on: April 14, 2015, 03:06:19 pm »
Strangely, I tried negating final position y and z in shader (which i think it's same with rotateX) and it works ok now. I wonder if I missed somewhere in software part.