Test #1

AxonSpark

Hey all, sorry I've been away so long. I've been swamped with freelance work and have been trying to align schedules with test subjects. Unfortunately, my first test was unsuccessful, but I'm going to lay the suit out and try initializing the biped configuration to see if I can find the problem. Here's what happened:

Once I got everything connected correctly (having the K-Ceptors not light up if they are incorrectly connected was a nice bit of the design that I found helpful), I ran the following from notochord/bin:

./notochord -c ../default_byped.xml <CLIENT COMPUTER IP ADDRESS>

In some instances, I got this error message:

In other cases, the K-Ceptors started initializing, but the dorsal and head would fail, leading to an infinite loop of errors. I thought part of this might have been due to improperly wiring the K-Ceptors to the Hub (which @daylanKifky specified here. I made sure the cables were plugged into the correct Hub channels, but was still getting errors.

So then I thought the cables might be the issue, and tried swapping what should've been the problematic cable with another, but then K-Ceptors connected to cables unrelated to the swap failed instead. I also double checked to make sure that all the cables were securely snapped into the K-Ceptors and the Hub and got the same results.

The last thing I noticed was that as my test subject was moving around during the errors, sometimes the K-Ceptors would spontaneously connect and disconnect. So at this point I'm thinking the cables themselves might be the issue, and I may need to purchase new ones.

Other notes:

I'm hoping the final production model has a suit or system of straps that are easier to deal with. The most frequent problem was slipping velcro straps (and this is with the hook side facing the test subject's clothing and the soft side facing outward).

The other issue is something I need to figure out. The project box I have for the Hub has a little too much height to it (it was the only one I found that could fit it at the time), so it's a challenge to fit that, the Raspberry Pi, and the battery pack into a modestly sized pouch. I'm also currently using a reflective jogging harness for the dorsal sensor and the belt that carries the pouch. It's a little on the flimsy side, but it's the best solution I could find currently, but I think the production version should have thicker, more substantial straps.

daylanKifky

Hi @AxonSpark, nice to see you are back!

The error you're getting is because the notochord can't find the XML on the specified path. In this particular case you are missing a backslash bettwen the name of the file (default_biped.xml) and the relative path (..). So the correct command is:

./notochord -c ../default_biped.xml 10.0.1.48

AxonSpark In other cases, the K-Ceptors started initializing, but the dorsal and head would fail,

I guess those K-Ceptors worked before when connected individually, right? It could be a problem with the soldering of the RJ12 connector of that gate. As a test you can plug the problematic K-Ceptors on another gate and run

./notochord --scan

to check if it can find them.

Then plug some K-Ceptors to the gate #2 and repeat the scan. If no error is raised, then try gently shaking the hardware or cables a little while you do it.

AxonSpark I'm hoping the final production model has a suit or system of straps that are easier to deal with.

We are working on the production version which will come with this, and many others improvements.

let me now how it goes!

AxonSpark

Hey daylanKifky,

So I think I found the problem. All the K-Ceptors worked beforehand, as they all worked the last time I calibrated them. Still I tested all of them, and each Hub channel using the scan command and a working cable.

Next, I went through and tested each cable, and found 2 that were faulty. One unreliably connected to the K-Ceptor and was affected by shaking, and sometimes returned this result:

The other cable unreliably powered the K-Ceptor and also was affected by shaking.

So it looks like I need to get two new 6-core RJ-12 crimped cables, correct?

daylanKifky

AxonSpark

Well that's good and bad news.

The short answer is: yes you should get a couple of new RJ12 6p6c crimped cables.
The issue is: they are not so easy to find, specially because the order of the wires should be mirrored, take a look here.

it's probably easier to cut off the existing headers, get a couple of new 6p6c RJ12 headers and crimp them your own. I know that sounds scary, but it's actually not a hard process, you will find plenty of videos online showing how it can be done. Otherwise, as usual, my advice is to ask for help on your local fablab or similar place.

I'm really sorry about this malfunction. We tested each cable manually before shipping, but they turned out to be somewhat fragile.

However..

In order to sidestep the problem you can (momentarily) leave 2 K-Ceptors out of the capture, for example those on the feet. You can use this XML for that.

In blender you will need to:

unselect the Capture bone property from the Chordata addon > Advanced (option only visible in pose mode)
select inherit rotation on the Bone properties panel > Relations

for each of the feet
Let me know how it turns out

AxonSpark

@daylanKifky

So I went and got some new 6p6c RJ12 headers and used the wiring from an Ethernet cable to make the new cables. The process went smoothly, and I was able to power up the whole suit and initialize the default biped configuration.

However, things went less smoothly inside Blender 2.79. Since I was using myself as the test subject, I stepped away from my laptop, faced south, used my wireless mouse to hit the 'Receive' button, and then went into a T-Pose which I maintained for several seconds to let the sensors settle. In order to get out of the calibration and not break the T-Pose, I used the voice dictation feature in OSX to hit the Return key when I issued a verbal command. Here's a video of the results:

Keep in mind that I was using Blender's internal screencast feature (need to find a better solution), and then slowed down the image sequence to 20% speed in Premiere, so the T-Pose is being held longer than it might seem in the video.

I'm not sure what the problem is, but the joints on the rig are not bending the same way I am (for instance lifting a leg and bending my knee forward results in completely different result in Blender, usually bending to the side, or sometimes backward), and even when the rig goes in roughly the right direction the bend doesn't correspond to the degree that I'm bending (eg., I'm being my arms a lot more than the rig is).

Any ideas on what might be going wrong? I know I last individually calibrated the K-Ceptors a while ago, but I would think that the calibration data would still be valid.

daylanKifky

AxonSpark

Hey
A shout-out to you for crimping the new RJ12 headers and for doing the one-man-band capture!! the voice command technique in quite ingenious!

The behaviour you are getting on the model looks a lot like the performer (in this case yourself, but let's just call it that way) was facing in the wrong direction. But as you mentioned, you were facing south... 🤔

I would take you sensors for correctly calibrated, even if you did it several months ago. You usually can tell when you are wearing a non calibrated sensor because it drifts even if you are completely steady. And it doesn't seems the case.

It's hard to tell what's going on from the video, but there's something that i'm having trouble understanding. During pose-calibration, the sensors on the back of the body normally(*) have their Z-axis pointing to the +X of the scene. And those on the legs should normally point to the -X on the scene. But during your calibration some of them were pointing in direction of the Y axis.

Do you have any idea of why is this happening?

(*) This doesn't have to necessarilly be like this. It depends on how the K-Ceptors are positioned on the body. This applies if the KCs on the back are resting on the flat over the spine area, and the ones on the legs are resting flat on the front of your legs.

daylanKifky

Hi!
I came out with a workflow to debug what's happening.

You can put together a simplified rigid "performer" to better see the orientation of the KCs in blender. Just use elastic bands or tape to fix three KCs to a rigid support (like a long ruler). You should also fix the SBC, Hub and battery.
All three KCs should be oriented roughly in the same direction

Then take a look at this video:

Files for this test:

Let me know how it goes.

PS: in the video we ask you to point the KCeptors north. This is because they are supposed to be on the back of our dummy performer, so it is pointing south

AxonSpark

Hi @daylanKifky,

Here are my test results:

So, I'm definitely having drifting/orientation issues (or certain axes being linked to incorrect corresponding axes). I'm guessing I'll have to re-calibrate, correct? If I do, I think I may need to update my software as well, since I'm currently running Notochord v0.1.1b.

I'm pretty sure I set up the spine rig correctly, but just to make sure, here's a picture of my set up:

As far as your question about K-Ceptors pointing along the wrong axes goes, I'm not sure why this is happening. Generally, I orient the K-Ceptors along the input/output lines so the ports from one K-Ceptor are pointed toward the next. Do they need to be oriented differently? As for the back, the bottom of the sensors faces towards the back, and the top of the sensors faces away from the back. I drew some info on your DefBypedConf-IDModules to try and illustrate some of this:

Let me know how I should proceed. Thanks!

Note for anyone else doing this test: your K-Ceptors need to be plugged into CH 1 on the Hub (unless you change the XML file).

daylanKifky

Yep, it looks like you have a sensor calibration problem there. I remember you were having good results on the last session, some months ago. That calibration should stay on the sensors, so I have no idea what's the cause of the current problem.

I think trying a re-calibration now is a good idea. I suggest you to take a set o three KCs, calibrate each one, and then use them on the spine test.

(There wasn't many visible changes since v0.1.1b, we are working on modifying some internal interfaces, but basically the functionality of the program is the same)

AxonSpark I'm pretty sure I set up the spine rig correctly

You did

AxonSpark I orient the K-Ceptors along the input/output lines so the ports from one K-Ceptor are pointed toward the next. Do they need to be oriented differently? As for the back, the bottom of the sensors faces towards the back, and the top of the sensors faces away from the back.

In technical terms, you can orient the KCs as you want, as far as they stay well-fixed to to body part they have to capture. In this case I was asking it in order to confirm if the differences in orientation I noticed in your first video were due to a sensor malfunction (perhaps you were just positioning them that way)

The arrangement you are describing is the standard one.

AxonSpark

Hey daylanKifky,

Just dropping a note to say that more test results are forthcoming. I did the K-Ceptor calibrations today, along with test cube and spine tests and things seem to be working. I also decided to do another full test, and things are generally bending in the right direction. However, the movements of the rig still don't fully correspond to what I'm doing and seem pretty stiff, and it seems to take a while for parts of the rig to get into a pose when I make one.

I have the graph images generated by each calibrated K-Ceptor, pictures taken of how I have the sensors arranged on the suit, and I did a simultaneous screen cap/video of the full biped test. All I have to do now is redo the spinal tests so I can record them. It might take me a day or two due to work stuff, but it's coming. :-)

In the meantime I made a few suggested changes to the Wiki.

daylanKifky

AxonSpark

Many thanks for taking your feedback so seriously! I approved your edits on the wiki. It was a good idea to let me know about your edits here: we receive an annoying amount of spam though wiki articles editing. So it hard to tell from legitimate edits at first sight.

AxonSpark it seems to take a while for parts of the rig to get into a pose when I make one.

Do you mean that you are getting a noticeable lag when using the full body, that is not present on smaller tests?

AxonSpark

daylanKifky Yes, there is a significant amount of lag in the full body that is not present in the smaller tests.

AxonSpark

Ok, here we go. I have a lot of material, so I'm going to parse it out into 3 posts. Here are the results from the latest round of K-Ceptor calibrations.

AxonSpark

After I calibrated the K-Ceptors, I tested each one with the test cube Blender file. All 15 seemed to be working properly. I also performed spine tests after the cube tests, and they too seemed to be giving me good results.

However, when I did the spine tests again today in order to record them, one of the K-Ceptors didn't seem to be working right. For some reason, K-Ceptor 1 in Group 4 had a constant drift that didn't occur on the previous test. Do you know what might have caused this? Since I re-calibrated and tested all the K-Ceptors 3 days ago, they should all be working correctly. I can't imagine what would have caused this one to malfunction (note: I did the test with Group 4 a couple times to confirm the result, and in both instance K-Ceptor 1 was drifting).

Here are the full results of the spinal tests (apologies for the framing of the video in the corner):

AxonSpark

Here is my second full biped test:

I did the test a couple times before filming/capturing and got the same results you see here. With my Attempt #1, the joints didn't bend in the right direction, but when I reloaded the Blend file and tried again, the joints would bend correctly.

The other big issue is some sort of lag. The full rig is nowhere near as responsive as the test cube or spinal test. When I move, the corresponding parts of the rig sometimes sort of go in the right direction, but the movements are much more subtle than the ones I'm making. It also seems to take the rig a long time to match my pose. For instance when I lifted my arms over my head, the arms on the rig gradually drifted up to match, and the rig arms weren't very responsive when I put my arms back down. It seems like rig is reacting in time with my movements, but the data is being applied in an odd, slow way.

A few things to note:

These issues were happening before doing the screen capture, so I don't think it was an issue of system resources.
At this point K-Ceptor 1 in Group 4 was working fine.
Despite some strap issues with the right thigh and left arm, the sensors were all in place when I started.

Also, to give you a better view of how I setup my suit, I had a friend take some pictures:

I think I'm ultimately going to have to get something better than this jogging harness. Do you happen to know what the testers in India have been using?

Lastly, I've noticed that the bone that corresponds with the base doesn't move (most noticeable when I was doing squats and push-ups) and everything sort of rotates around it. In the future will it be able to move around in 3D space?

daylanKifky

Hi!
thanks as always for your accurate report. It might take me a couple of days to process the information.

AxonSpark Do you happen to know what the testers in India have been using?

We will ask. Looks like a pretty neat solution

AxonSpark Lastly, I've noticed that the bone that corresponds with the base doesn't move (most noticeable when I was doing squats and push-ups) and everything sort of rotates around it. In the future will it be able to move around in 3D space?

Yes it will, in fact you can already export that information when re-transmiting your capture live by selecting the option Send root bone height. We will make it available on the Blender viewport as soon as possible.

AxonSpark I logged into the Wiki to add a section on connecting to your WiFi router for testers that might be new to the Raspberry Pi, but it says that I was blocked indefinitely in June. Not sure why.

Sorry about that, we receive spam accounts everyday and just block them all. It is hard to tell them from real ones, since it's an automated process. It should be working now

AxonSpark

P.S. I logged into the Wiki to add a section on connecting to your WiFi router for testers that might be new to the Raspberry Pi, but it says that I was blocked indefinitely in June. Not sure why.

daylanKifky

Hi @AxonSpark
Sorry for the delay on my response. The behavior you are getting is inconsistent, and contradictory (i guess you already figured out that part 😅), so I needed some time to carefully go though it.
I'm still not able to give you a definitive response, but ..

Let's review what we can assume at this point.

The K-Ceptors seems to be working much better on your spine test 1.2, the behaviour you are getting once you enter capture mode is coherent.

The behavior on the second attempt of your full body test seems to be correct, in the sense that the joints are bending on the right direction. But the lag is huge, so we can't be 100% sure of this statement.

The origin of the lag is a complete mystery. You were using the exact same hardware, software, and network to make the spinal tests (I assume) and they presented no lag. Some probable causes i can think of are:

Network congestion: The amount of data coming from the Chordata system is not much, but under certain conditions the packets might accumulate causing significant delays, specially under wifi. This is very unlikely to happen, specially if you are using a decent access point. But just to be sure: Could you test it with a LAN cable instead of the WIFI? When doing this be sure to turn off the WIFI connection on your raspberry or router (not only your pc), otherwise you might end up transmitting over wifi even if the LAN cable is connected.
Pose-processing delays: This is a (again) a very unlikely scenario. Your computer is probably able to handle the processing of the capture, the fact that the spinal test runs smoothly confirms that. Both files are equivalent in terms of bone and poly count. But perhaps there's something on the configuration of the default_biped file that's causing the lag. An example would be debug mode turned on. On that mode Blender prints every received message to the console, which is a blocking operation. Try turning everything but the strict necessary off under Record, Transmission and Advanced panel.

The still open questions are:

Why the default_biped test worked only after reloading the file? Have you noticed something similar before? This looks like a bug on the add-on. It would be great if we can reproduce the exact situation were it happens. What did you did on Blender before you made that attempt?
Why the K-Ceptors lost their calibration? I assume they were stored properly. Perhaps the temperature on your flat is now drastically different from the time you first calibrated them?
What's happening with K-Ceptor 1 in Group 4? That intermittent behavior you are describing seems to indicate some kind of bad contact on a component on the KCeptor. The U1 and U2 shouldn't be involved (otherwise you would get a communication error) but perhaps there's a badly soldered pad on some of the capacitors around U1.
Another possible bad soldering might be some pad on U3 (the memory). If you ever get this behavior again try running notochord with a higher verbosity level, in order for it to print warnings (such as a memory not found)

PS: we still had no answer from the betatesters in india. @robin is that you on the photo @AxonSpark is talking about?

AxonSpark

Hey @daylanKifky,

I completely understand the delay. Between Maker Faires, the upcoming Blender conference, and developing new K-Ceptors, this is probably a busy month for the Chordata team. To answer your quesitons:

daylanKifky You were using the exact same hardware, software, and network to make the spinal tests (I assume) and they presented no lag

Yes, everything was the same.

Network Congestion

I also think this is unlikely, as in the past 4 people have shared the connection on my Asus Dark Knight router without problems. And when I run the tests I’m alone in my apartment, so I’m the only one using it. I have some freelance editing work this week, but I’ll get the required cables to test the default biped configuration over Ethernet as soon as I’m able.

Pose-processing delays

I’m not using debug mode. Although, here’s a screenshot of the panels as they appear after I load the default_biped blend file (I haven’t touched these when doing testing. Just the Connect and Receive buttons):

Also, I’m currently using 2.79b if that helps.

daylanKifky Why the default_biped test worked only after reloading the file? Have you noticed something similar before? This looks like a bug on the add-on. It would be great if we can reproduce the exact situation were it happens. What did you did on Blender before you made that attempt?

I have not noticed anything similar before, and I didn’t do anything special in Blender. My order of operations is this:

Turn on the power and give my Raspberry Pi a moment to boot up
Connect to the Pi via the MacOS terminal using SSH
Run Notochord with the default_biped.xml file
Open up default biped blend file and connect to the Pi
Do my wireless mouse/voice command combo to start the T-Pose calibration and capture
The joints don’t bend correctly, so I reload the default_biped blender file again, and for some reason it works

daylanKifky Why the K-Ceptors lost their calibration? I assume they were stored properly. Perhaps the temperature on your flat is now drastically different from the time you first calibrated them?

What is considered proper storage? Since the suit has a lot of parts, I store everything in a small canvas duffel bag that sits on my plastic lighting case on my equipment shelf which is made of metal. Also, the temperature in my apartment hasn’t drastically changed. Athough, I should ask: what sort of temperature fluctuation would cause the calibration to fail?

daylanKifky What's happening with K-Ceptor 1 in Group 4? That intermittent behavior you are describing seems to indicate some kind of bad contact on a component on the KCeptor. The U1 and U2 shouldn't be involved (otherwise you would get a communication error) but perhaps there's a badly soldered pad on some of the capacitors around U1.
Another possible bad soldering might be some pad on U3 (the memory). If you ever get this behavior again try running notochord with a higher verbosity level, in order for it to print warnings (such as a memory not found)

I looked at the connections under a magnifying glass, and everything seems to be soldered well. Here is a screen shot of me testing this K-Ceptor in verbose mode:

Hope some of this helps.

daylanKifky

AxonSpark

yeah you are right, this is a really busy month. We are also preparing the production for the second beta testing program.

Regarding your situation: As I supposed you do have a good router, so the network congestion hypothesis is even more unlikely. Still trying the Ethernet cable will help discard the WIFI factor.

It seems like your configuration is clean.

Is the procedure you are describing exactly the same each time? Do you have to necessarily repeat this step each time?

AxonSpark The joints don’t bend correctly, so I reload the default_biped blender file again, and for some reason it works

If that's the case then something really strange is happening. But at least we have identified a procedure to reproduce it.

From the dump you are sending it looks like the calibration stored on the EEPROM was correctly read. (you can tell by the word EEPROM next to each calibration vector/matrix. I guess it was working fine that time.

I guess at this point the only thing we can try is to expand the spinal test, to see how a slightly bigger set of sensors behave. For example you can build an expanded rigid performer with which you can test the functionality of the joints more comfortably. Something like this:

What do you think? You will need a different .blend and XML for that, I'll send them as soon as possible