Cluster Help

[videcuk]

Hi

We have just recently purchased Vizard Enterprise and are currently setting it up to work on our simulation centre which has one master PC and three image generators on a LAN to project onto a large screen. The cluster master and client utilities seem to be working fine. The master finds each of the clients through the vizard utility and is pinging them with no problem. We have the files and resources we want to run on a shared network drive so all the clients and the master can access this. As a test we amended a fairly simple file from the vizard examples as per the cluster section in the manual, including the pushmask and popmask to Clients1 - 4.

However, when we try and run a simulation we get the following error messages:

On the master PC we get the following error in the Vizard console:
**NOTIFY: Launching in cluster master mode. Waiting for 3 client(s) to connect.
**ERROR: VizMultiCastTCP::init( ) failed to connect to following clients:
Image-Gen-1
**ERROR: VizMultiCastTCP::init( ) failed to connect to following clients:
Image-Gen-2
**ERROR: VizMultiCastTCP::init( ) failed to connect to following clients:
Image-Gen-3
**ERROR: Failed to establish connection with all clients
Loading File: male.cfg
**Load Time: 0.38 seconds

On each of the client PC we get the following in the console window:
** Received ping from VR-MASTER
** Received ping from VR-MASTER
** Received ping from VR-MASTER
** Launching Y:\motions1.py from VR-MASTER
************************************************** ****************************
** Loading motions1.py
************************************************** ****************************
** NOTIFY: Launching in cluster client mode with ID 2
** ERROR: VizReceiveTCP::init() failed to accept connection (WSAEWOULDBLOCK)
** ERROR: Failed to establish connection with master

Not sure if anyone can help with this one, but would really appreciate any advice as we're still pretty new to Vizard.

Many thanks

[farshizzo]

Hi,

It seems that the clients are not receiving the connection request from the master. Do you have Windows Firewall enabled?

[videcuk]

Hi
We have disabled all of the firewalls on the master PC and all of the clients in the cluster but still get the same message...

[farshizzo]

Hi,

Is the error message printed out immediately or after 15 seconds? Also, can you try something. In the interactive window of the master computer enter the following commands:

>>> import socket
>>> socket.gethostbyname('Image-Gen-1')

How long does it take to print out something, and what does it print out?

[videcuk]

Hi

The error on the master PC is printed out after around 15 seconds. The error on each of the clients follows ahortly after that.

We found an older thread on extending the client timeout using the 'vizcore.cfg' files and have extended the timeout on each of the clients to 100000 using 'viz.cluster.client.init_timeout = 100000' but still get the same errors.

When we put in the import socket etc in the interactive window as above, vizard returns the IP address (for example 134.220.14.241 for the PC named 'Image-Gen-1') of each of the clients immeadiately.

Thanks...

[farshizzo]

Hi,

This is a strange problem. Have you tried connecting to only 1 client instead of 3? Also, what version of Windows are you running?

[hotspur1]

I've got a similar thing with just a master and client. I get the following error

** NOTIFY: Launching in cluster master mode. Waiting for 1 client(s) to connect.
** ERROR: VizMultiCastTCP::init() failed to connect to following clients:
MONOLITH
** ERROR: Failed to establish connection with all clients

I've tried increasing the timeout values using vizcore.cfg to 30 sec, but that didn't help. I've turned off the firewall on both machines. Everything also seems to be fine when I start the Master and Client apps before running my simulation. The ping from master to client pops up right away.

Any suggestions on things to check?

Thanks

[farshizzo]

What does the client machine print out?

[hotspur1]

The client doesn't print anything out when the script runs. When I hit the refresh button on the master window, then the client prints a message that it has been pinged.

[farshizzo]

Does the cluster client print a message similar to the following:

** Launching myscript.py

If not try running the following commands in the Vizard interactive window from the master computer:

>>> import socket
>>> socket.gethostbyname('client_computer_name')

Replace client_computer_name with the name of the client computer. This command should print out the IP address of the client computer. Make sure this matches the client computers actual IP address.

[farshizzo]

Were you able to get the cluster working?

[hotspur1]

Yes, it did turn out to be a problem with the IP addresses and machine names. The last diagnostic provided the vital clues.

Thanks

[farshizzo]

Thanks for the update. We've made some changes in the next version of Vizard that will hopefully address this issue.