kinect + avatars

[Darkmax]

How can i link my kinect skeleton with an avatar, i tried with animating the bones tutorial, but the problem is that i have more bones than sensors values from the kinect skeleton so when i link the values of the kinect skeleton the other parts that i didn't link will remain static, so how can i resolve this?

[Jeff]

Our supported workflow for animating a character with tracking data is through Motionbuilder and Live Characters. Motionbuilder does all the calculations and determines where the bones should be using inverse kinematics.

[jkelly]

I, too, was hoping to use the Kinect to animate an avatar, but am dealing with the same problem of having more bones than sensors. I am using FAAST to acquire the sensor data, as described here: http://kb.worldviz.com/articles/1584.

The Kinect presents a potentially low cost option for motion tracking, and using Motionbuilder and Live Characters greatly increases the cost. Is there were a low cost way to animate an avatar using the Kinect? Thanks.

[Deltcho]

I recommend using the official microsoft SDK for the Kinect (it's free). The Kinect tracks very specific points on the body and you need to interpolate the position of the body from those points. The "Skeletal Tracker" demo that comes with the SDK should be all you need to get going. Just integrate a vrpn server into the code (and remove the visualization stuff to improve performance) and you're ready to go.

Or, you can calculate the position of each limb in euler coordinates within vizard using FAAST, but python is significantly slower than c#/c++ so I'd recommend doing it within the SDK framework and just sending the limb position over VRPN to Vizard.

When you've got the coordinates for the appropriate avatar limbs, you just link them within Vizard. It's a shame that Vizard can't work with the Kinect out of the box using official drivers.

[tobin]

Thanks for the comments about the MS Research SDK. The lead developer of the FAAST project, Evan Suma, offers the summary below of the pros and cons of the MS Research SDK and the OpenNI SDK (note that FAAST currently uses OpenNI but will soon add support for MS Research SDK thus giving users the ability to choose):

"OpenNI advantages:
- More accurate and robust for arm/leg pose.
- Better depth map resolution (always 640x480). When the skeleton tracker is activated with Microsoft, the depth map drops to 320x240.
- More permissive license that allows corporate use. The Microsoft SDK can only be used for research and noncommercial purposes.
- More stable. The Microsoft SDK is a little more buggy and unstable, and crashes for me every now and then.

Microsoft advantages:
- No calibration required. However, the word from PrimeSense is that they are very close to solving the calibration issue, and hopefully soon OpenNI will no longer require the calibration pose at all.
- More skeleton joints. With Microsoft you get an articulated neck joint, which is rigid with OpenNI. You also get the wrist and ankle joints, but in my experience these don't work very well.
- More sophisticated built-in filtering (to reduce noise and jitter).
"

[rlh001]

I tried this web page http://kb.worldviz.com/articles/1584 but it says The page you are looking for no longer exists. Where can i look now? Thanks

[tobin]

We are sorry about the inconvenience--we are in the process of streamlining the installation process and installation package. The link is alive again but we encourage you to wait until April 5 and take advantage of our update that will be posted then.

[Gladsomebeast]

Have some rough code here that moves the limbs and root/pelvis of the original Complete Character samples in Vizard with the kinect.

import viz
import vizact
import vizshape
import math

viz.go(viz.FULLSCREEN)

#viz.clearcolor(viz.SKYBLUE)
ground = viz.add('gallery.osgb')

#Kinect Tracker object ID's
#myHead = vrpn.addTracker( 'Tracker0@localhost', HEAD).
HEAD = 0
NECK = 1
TORSO = 2
WAIST = 3
LEFTCOLLAR = 4
LEFTSHOULDER = 5
LEFTELBOW = 6
LEFTWRIST = 7
LEFTHAND = 8
LEFTFINGERTIP = 9
RIGHTCOLLAR = 10
RIGHTSHOULDER = 11
RIGHTELBOW = 12
RIGHTWRIST = 13
RIGHTHAND = 14
RIGHTFINGERTIP = 15
LEFTHIP = 16
LEFTKNEE = 17
LEFTANKLE = 18
LEFTFOOT = 19
RIGHTHIP = 20
RIGHTKNEE = 21
RIGHTANKLE = 22
RIGHTFOOT = 23

#store trackers, links, and vizshape objects
trackers = []
links = []
shapes = []

#start vrpn
vrpn = viz.addExtension('vrpn7.dle')

#set the camera facing the avatar
viz.MainView.setPosition(0,0,0)

rootRaw = viz.addGroup(euler=[0, 0, 0])
lastX = 0
lastZ = 3
#now add all trackers and link a shape to it
for i in range(0,24):
	t = vrpn.addTracker( 'Tracker0@localhost',i )
	
	s = viz.add('wheelbarrow.ive', scale=[.3]*3, parent=rootRaw)
	if i == TORSO:
		s.color(viz.RED)
		#display an arrow pointing away from the avatar		
	elif i in [LEFTSHOULDER,RIGHTSHOULDER,LEFTHIP,RIGHTHIP]:
		s.color(viz.BLUE)

	l = viz.link(t,s)
	trackers.append(t)
	links.append(l)
	shapes.append(s)

#Debug 3d models with orientaton affordances
rootWheelbarrow = viz.add('wheelbarrow.ive', scale=[.3]*3)
rootWheelbarrow.color(viz.YELLOW)
target = viz.add('wheelbarrow.ive', scale=[.3]*3, pos=[.5, 0, 0])
target.color(viz.GREEN)

#girl = viz.add('vcc_female.cfg', pos=[0, -1, 0], euler=[180, 0, 0])
girl = viz.add('vcc_male.cfg', pos=[0, 0, 0], euler=[0, 0, 0])

RF = vrpn.addTracker('Tracker0@localhost', RIGHTFOOT)
LF = vrpn.addTracker('Tracker0@localhost', LEFTFOOT)
RK = vrpn.addTracker('Tracker0@localhost', RIGHTKNEE)
LK = vrpn.addTracker('Tracker0@localhost', LEFTKNEE)
Torso = vrpn.addTracker('Tracker0@localhost', TORSO)
head = vrpn.addTracker('Tracker0@localhost', HEAD)
neck = vrpn.addTracker('Tracker0@localhost', NECK)
rightShoulder = vrpn.addTracker('Tracker0@localhost', RIGHTSHOULDER)
rightElbow = vrpn.addTracker('Tracker0@localhost', RIGHTELBOW)
leftShoulder = vrpn.addTracker('Tracker0@localhost', LEFTSHOULDER)
leftElbow = vrpn.addTracker('Tracker0@localhost',LEFTELBOW)
rightHip = vrpn.addTracker('Tracker0@localhost',RIGHTHIP)
rightKnee = vrpn.addTracker('Tracker0@localhost',RIGHTKNEE)
leftHip = vrpn.addTracker('Tracker0@localhost',LEFTHIP)
leftKnee = vrpn.addTracker('Tracker0@localhost',LEFTKNEE)

waist = vrpn.addTracker('Tracker0@localhost', WAIST)
rootWaist = girl.getBone('Bip01')
rootWaist.lock()

headBone = girl.getBone('Bip01 Head')
headBone.lock()

kinectToCharacter = {	'Bip01 L UpperArm':(rightShoulder, [0, 90, 0]), \
						'Bip01 L Forearm':(rightElbow, [0, 90, 0]), \
						'Bip01 R UpperArm':(leftShoulder, [0, -90, 180]), \
						'Bip01 R Forearm':(leftElbow, [0, -90, 180]), \
						'Bip01 L Thigh':(rightHip, [90, 0, -90]), \
						'Bip01 L Calf':(rightKnee, [90, 0, -90]), \
						'Bip01 R Thigh':(leftHip, [90, 0, -90]), \
						'Bip01 R Calf':(leftKnee, [90, 0, -90]), \
						'Bip01':(waist, [0, 90, -90]) \
					}
bones = {}
for boneName in kinectToCharacter:
	b = girl.getBone(boneName)
	b.lock()
	bones[b] = kinectToCharacter[boneName]

boneToTest = girl.getBone('Bip01 L Thigh')
boneToTest.lock()
boneToTest.setEuler([180, 0, 0])
import vizmat
def animate():
	#debug code
	m = vizmat.Transform()
	euler = rightHip.getEuler()
	m.setEuler(euler)
	m.preEuler([90, 0, -90])
	
	target.setEuler(m.getEuler())
	rootWheelbarrow.setEuler( boneToTest.getEuler(viz.ABS_GLOBAL) )
#	rootWaist.setEuler(m.getEuler(), viz.ABS_GLOBAL)
	
	#position of root
	pos = waist.getPosition()
	rootWaist.setPosition(pos, viz.ABS_GLOBAL)
#	rootWaist.setPosition([2.64658, 0.53107, 3.37627], viz.ABS_GLOBAL) #if you want to fix the avatar to point in room.
	
	for bone in bones:
		kinectTracker = bones[bone][0]
		euler = kinectTracker.getEuler()
		m = vizmat.Transform()
		m.setEuler(euler)
		m.preEuler(bones[bone][1])
		bone.setEuler(m.getEuler(), viz.ABS_GLOBAL)
	
vizact.ontimer(0, animate)

viz.MainView.setPosition([0, 0, 5])
viz.MainView.setEuler([180, 0, 0])

import vizcam
viz.cam.setHandler(vizcam.PivotNavigate(distance=5))

def viewInHead():
	m = vizmat.Transform()
	m.setEuler(headBone.getEuler(viz.ABS_GLOBAL))
	m.preEuler([-90, -90, 0])
	viz.MainView.setEuler( m.getEuler() )
	viz.MainView.setPosition( headBone.getPosition(viz.ABS_GLOBAL) )

#vizact.ontimer(0, viewInHead)